mirror of
https://git.datalinker.icu/comfyanonymous/ComfyUI
synced 2026-11-21 19:27:09 +08:00
requirements
This commit is contained in:
commit
4ddad6ea17
48
.github/ISSUE_TEMPLATE/bug-report.yml
vendored
Normal file
48
.github/ISSUE_TEMPLATE/bug-report.yml
vendored
Normal file
@ -0,0 +1,48 @@
|
|||||||
|
name: Bug Report
|
||||||
|
description: "Something is broken inside of ComfyUI. (Do not use this if you're just having issues and need help, or if the issue relates to a custom node)"
|
||||||
|
labels: ["Potential Bug"]
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
attributes:
|
||||||
|
value: |
|
||||||
|
Before submitting a **Bug Report**, please ensure the following:
|
||||||
|
|
||||||
|
- **1:** You are running the latest version of ComfyUI.
|
||||||
|
- **2:** You have looked at the existing bug reports and made sure this isn't already reported.
|
||||||
|
- **3:** You confirmed that the bug is not caused by a custom node. You can disable all custom nodes by passing
|
||||||
|
`--disable-all-custom-nodes` command line argument.
|
||||||
|
- **4:** This is an actual bug in ComfyUI, not just a support question. A bug is when you can specify exact
|
||||||
|
steps to replicate what went wrong and others will be able to repeat your steps and see the same issue happen.
|
||||||
|
|
||||||
|
If unsure, ask on the [ComfyUI Matrix Space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) or the [Comfy Org Discord](https://discord.gg/comfyorg) first.
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Expected Behavior
|
||||||
|
description: "What you expected to happen."
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Actual Behavior
|
||||||
|
description: "What actually happened. Please include a screenshot of the issue if possible."
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Steps to Reproduce
|
||||||
|
description: "Describe how to reproduce the issue. Please be sure to attach a workflow JSON or PNG, ideally one that doesn't require custom nodes to test. If the bug open happens when certain custom nodes are used, most likely that custom node is what has the bug rather than ComfyUI, in which case it should be reported to the node's author."
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Debug Logs
|
||||||
|
description: "Please copy the output from your terminal logs here."
|
||||||
|
render: powershell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Other
|
||||||
|
description: "Any other additional information you think might be helpful."
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
8
.github/ISSUE_TEMPLATE/config.yml
vendored
Normal file
8
.github/ISSUE_TEMPLATE/config.yml
vendored
Normal file
@ -0,0 +1,8 @@
|
|||||||
|
blank_issues_enabled: true
|
||||||
|
contact_links:
|
||||||
|
- name: ComfyUI Matrix Space
|
||||||
|
url: https://app.element.io/#/room/%23comfyui_space%3Amatrix.org
|
||||||
|
about: The ComfyUI Matrix Space is available for support and general discussion related to ComfyUI (Matrix is like Discord but open source).
|
||||||
|
- name: Comfy Org Discord
|
||||||
|
url: https://discord.gg/comfyorg
|
||||||
|
about: The Comfy Org Discord is available for support and general discussion related to ComfyUI.
|
||||||
32
.github/ISSUE_TEMPLATE/feature-request.yml
vendored
Normal file
32
.github/ISSUE_TEMPLATE/feature-request.yml
vendored
Normal file
@ -0,0 +1,32 @@
|
|||||||
|
name: Feature Request
|
||||||
|
description: "You have an idea for something new you would like to see added to ComfyUI's core."
|
||||||
|
labels: [ "Feature" ]
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
attributes:
|
||||||
|
value: |
|
||||||
|
Before submitting a **Feature Request**, please ensure the following:
|
||||||
|
|
||||||
|
**1:** You are running the latest version of ComfyUI.
|
||||||
|
**2:** You have looked to make sure there is not already a feature that does what you need, and there is not already a Feature Request listed for the same idea.
|
||||||
|
**3:** This is something that makes sense to add to ComfyUI Core, and wouldn't make more sense as a custom node.
|
||||||
|
|
||||||
|
If unsure, ask on the [ComfyUI Matrix Space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) or the [Comfy Org Discord](https://discord.gg/comfyorg) first.
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Feature Idea
|
||||||
|
description: "Describe the feature you want to see."
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Existing Solutions
|
||||||
|
description: "Please search through available custom nodes / extensions to see if there are existing custom solutions for this. If so, please link the options you found here as a reference."
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Other
|
||||||
|
description: "Any other additional information you think might be helpful."
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
32
.github/ISSUE_TEMPLATE/user-support.yml
vendored
Normal file
32
.github/ISSUE_TEMPLATE/user-support.yml
vendored
Normal file
@ -0,0 +1,32 @@
|
|||||||
|
name: User Support
|
||||||
|
description: "Use this if you need help with something, or you're experiencing an issue."
|
||||||
|
labels: [ "User Support" ]
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
attributes:
|
||||||
|
value: |
|
||||||
|
Before submitting a **User Report** issue, please ensure the following:
|
||||||
|
|
||||||
|
**1:** You are running the latest version of ComfyUI.
|
||||||
|
**2:** You have made an effort to find public answers to your question before asking here. In other words, you googled it first, and scrolled through recent help topics.
|
||||||
|
|
||||||
|
If unsure, ask on the [ComfyUI Matrix Space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) or the [Comfy Org Discord](https://discord.gg/comfyorg) first.
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Your question
|
||||||
|
description: "Post your question here. Please be as detailed as possible."
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Logs
|
||||||
|
description: "If your question relates to an issue you're experiencing, please go to `Server` -> `Logs` -> potentially set `View Type` to `Debug` as well, then copypaste all the text into here."
|
||||||
|
render: powershell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
attributes:
|
||||||
|
label: Other
|
||||||
|
description: "Any other additional information you think might be helpful."
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
23
.github/workflows/pylint.yml
vendored
Normal file
23
.github/workflows/pylint.yml
vendored
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
name: Python Linting
|
||||||
|
|
||||||
|
on: [push, pull_request]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
pylint:
|
||||||
|
name: Run Pylint
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout repository
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@v2
|
||||||
|
with:
|
||||||
|
python-version: 3.x
|
||||||
|
|
||||||
|
- name: Install Pylint
|
||||||
|
run: pip install pylint
|
||||||
|
|
||||||
|
- name: Run Pylint
|
||||||
|
run: pylint --rcfile=.pylintrc $(find . -type f -name "*.py")
|
||||||
110
.github/workflows/stable-release.yml
vendored
Normal file
110
.github/workflows/stable-release.yml
vendored
Normal file
@ -0,0 +1,110 @@
|
|||||||
|
|
||||||
|
name: "Release Stable Version"
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- 'v*'
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
package_comfy_windows:
|
||||||
|
permissions:
|
||||||
|
contents: "write"
|
||||||
|
packages: "write"
|
||||||
|
pull-requests: "read"
|
||||||
|
runs-on: windows-latest
|
||||||
|
strategy:
|
||||||
|
matrix:
|
||||||
|
python_version: [3.11.8]
|
||||||
|
cuda_version: [121]
|
||||||
|
steps:
|
||||||
|
- name: Calculate Minor Version
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
# Extract the minor version from the Python version
|
||||||
|
MINOR_VERSION=$(echo "${{ matrix.python_version }}" | cut -d'.' -f2)
|
||||||
|
echo "MINOR_VERSION=$MINOR_VERSION" >> $GITHUB_ENV
|
||||||
|
- name: Setup Python
|
||||||
|
uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: ${{ matrix.python_version }}
|
||||||
|
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
persist-credentials: false
|
||||||
|
- shell: bash
|
||||||
|
run: |
|
||||||
|
echo "@echo off
|
||||||
|
call update_comfyui.bat nopause
|
||||||
|
echo -
|
||||||
|
echo This will try to update pytorch and all python dependencies.
|
||||||
|
echo -
|
||||||
|
echo If you just want to update normally, close this and run update_comfyui.bat instead.
|
||||||
|
echo -
|
||||||
|
pause
|
||||||
|
..\python_embeded\python.exe -s -m pip install --upgrade torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu${{ matrix.cuda_version }} -r ../ComfyUI/requirements.txt pygit2
|
||||||
|
pause" > update_comfyui_and_python_dependencies.bat
|
||||||
|
|
||||||
|
python -m pip wheel --no-cache-dir torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu${{ matrix.cuda_version }} -r requirements.txt pygit2 -w ./temp_wheel_dir
|
||||||
|
python -m pip install --no-cache-dir ./temp_wheel_dir/*
|
||||||
|
echo installed basic
|
||||||
|
ls -lah temp_wheel_dir
|
||||||
|
mv temp_wheel_dir cu${{ matrix.cuda_version }}_python_deps
|
||||||
|
mv cu${{ matrix.cuda_version }}_python_deps ../
|
||||||
|
mv update_comfyui_and_python_dependencies.bat ../
|
||||||
|
cd ..
|
||||||
|
pwd
|
||||||
|
ls
|
||||||
|
|
||||||
|
cp -r ComfyUI ComfyUI_copy
|
||||||
|
curl https://www.python.org/ftp/python/${{ matrix.python_version }}/python-${{ matrix.python_version }}-embed-amd64.zip -o python_embeded.zip
|
||||||
|
unzip python_embeded.zip -d python_embeded
|
||||||
|
cd python_embeded
|
||||||
|
echo ${{ env.MINOR_VERSION }}
|
||||||
|
echo 'import site' >> ./python3${{ env.MINOR_VERSION }}._pth
|
||||||
|
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
|
||||||
|
./python.exe get-pip.py
|
||||||
|
./python.exe --version
|
||||||
|
echo "Pip version:"
|
||||||
|
./python.exe -m pip --version
|
||||||
|
|
||||||
|
set PATH=$PWD/Scripts:$PATH
|
||||||
|
echo $PATH
|
||||||
|
./python.exe -s -m pip install ../cu${{ matrix.cuda_version }}_python_deps/*
|
||||||
|
sed -i '1i../ComfyUI' ./python3${{ env.MINOR_VERSION }}._pth
|
||||||
|
cd ..
|
||||||
|
|
||||||
|
git clone https://github.com/comfyanonymous/taesd
|
||||||
|
cp taesd/*.pth ./ComfyUI_copy/models/vae_approx/
|
||||||
|
|
||||||
|
mkdir ComfyUI_windows_portable
|
||||||
|
mv python_embeded ComfyUI_windows_portable
|
||||||
|
mv ComfyUI_copy ComfyUI_windows_portable/ComfyUI
|
||||||
|
|
||||||
|
cd ComfyUI_windows_portable
|
||||||
|
|
||||||
|
mkdir update
|
||||||
|
cp -r ComfyUI/.ci/update_windows/* ./update/
|
||||||
|
cp -r ComfyUI/.ci/windows_base_files/* ./
|
||||||
|
cp ../update_comfyui_and_python_dependencies.bat ./update/
|
||||||
|
|
||||||
|
cd ..
|
||||||
|
|
||||||
|
"C:\Program Files\7-Zip\7z.exe" a -t7z -m0=lzma2 -mx=8 -mfb=64 -md=32m -ms=on -mf=BCJ2 ComfyUI_windows_portable.7z ComfyUI_windows_portable
|
||||||
|
mv ComfyUI_windows_portable.7z ComfyUI/ComfyUI_windows_portable_nvidia.7z
|
||||||
|
|
||||||
|
cd ComfyUI_windows_portable
|
||||||
|
python_embeded/python.exe -s ComfyUI/main.py --quick-test-for-ci --cpu
|
||||||
|
|
||||||
|
ls
|
||||||
|
|
||||||
|
- name: Upload binaries to release
|
||||||
|
uses: svenstaro/upload-release-action@v2
|
||||||
|
with:
|
||||||
|
repo_token: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
file: ComfyUI_windows_portable_nvidia.7z
|
||||||
|
tag: ${{ github.ref }}
|
||||||
|
overwrite: true
|
||||||
|
prerelease: true
|
||||||
|
make_latest: false
|
||||||
76
.github/workflows/test-browser.yml
vendored
Normal file
76
.github/workflows/test-browser.yml
vendored
Normal file
@ -0,0 +1,76 @@
|
|||||||
|
# This is a temporary action during frontend TS migration.
|
||||||
|
# This file should be removed after TS migration is completed.
|
||||||
|
# The browser test is here to ensure TS repo is working the same way as the
|
||||||
|
# current JS code.
|
||||||
|
# If you are adding UI feature, please sync your changes to the TS repo:
|
||||||
|
# huchenlei/ComfyUI_frontend and update test expectation files accordingly.
|
||||||
|
name: Playwright Browser Tests CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [ main, master ]
|
||||||
|
pull_request:
|
||||||
|
branches: [ main, master ]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout ComfyUI
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
repository: "comfyanonymous/ComfyUI"
|
||||||
|
path: "ComfyUI"
|
||||||
|
- name: Checkout ComfyUI_frontend
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
repository: "huchenlei/ComfyUI_frontend"
|
||||||
|
path: "ComfyUI_frontend"
|
||||||
|
ref: "fcc54d803e5b6a9b08a462a1d94899318c96dcbb"
|
||||||
|
- uses: actions/setup-node@v3
|
||||||
|
with:
|
||||||
|
node-version: lts/*
|
||||||
|
- uses: actions/setup-python@v4
|
||||||
|
with:
|
||||||
|
python-version: '3.10'
|
||||||
|
- name: Install requirements
|
||||||
|
run: |
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||||
|
pip install -r requirements.txt
|
||||||
|
pip install wait-for-it
|
||||||
|
working-directory: ComfyUI
|
||||||
|
- name: Start ComfyUI server
|
||||||
|
run: |
|
||||||
|
python main.py --cpu 2>&1 | tee console_output.log &
|
||||||
|
wait-for-it --service 127.0.0.1:8188 -t 600
|
||||||
|
working-directory: ComfyUI
|
||||||
|
- name: Install ComfyUI_frontend dependencies
|
||||||
|
run: |
|
||||||
|
npm ci
|
||||||
|
working-directory: ComfyUI_frontend
|
||||||
|
- name: Install Playwright Browsers
|
||||||
|
run: npx playwright install --with-deps
|
||||||
|
working-directory: ComfyUI_frontend
|
||||||
|
- name: Run Playwright tests
|
||||||
|
run: npx playwright test
|
||||||
|
working-directory: ComfyUI_frontend
|
||||||
|
- name: Check for unhandled exceptions in server log
|
||||||
|
run: |
|
||||||
|
if grep -qE "Exception|Error" console_output.log; then
|
||||||
|
echo "Unhandled exception/error found in server log."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
working-directory: ComfyUI
|
||||||
|
- uses: actions/upload-artifact@v4
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: playwright-report
|
||||||
|
path: ComfyUI_frontend/playwright-report/
|
||||||
|
retention-days: 30
|
||||||
|
- uses: actions/upload-artifact@v4
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: console-output
|
||||||
|
path: ComfyUI/console_output.log
|
||||||
|
retention-days: 30
|
||||||
4
.github/workflows/test-ui.yaml
vendored
4
.github/workflows/test-ui.yaml
vendored
@ -24,3 +24,7 @@ jobs:
|
|||||||
npm run test:generate
|
npm run test:generate
|
||||||
npm test -- --verbose
|
npm test -- --verbose
|
||||||
working-directory: ./tests-ui
|
working-directory: ./tests-ui
|
||||||
|
- name: Run Unit Tests
|
||||||
|
run: |
|
||||||
|
pip install -r tests-unit/requirements.txt
|
||||||
|
python -m pytest tests-unit
|
||||||
|
|||||||
@ -33,8 +33,8 @@ jobs:
|
|||||||
build_dependencies:
|
build_dependencies:
|
||||||
runs-on: windows-latest
|
runs-on: windows-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v3
|
- uses: actions/checkout@v4
|
||||||
- uses: actions/setup-python@v4
|
- uses: actions/setup-python@v5
|
||||||
with:
|
with:
|
||||||
python-version: 3.${{ inputs.python_minor }}.${{ inputs.python_patch }}
|
python-version: 3.${{ inputs.python_minor }}.${{ inputs.python_patch }}
|
||||||
|
|
||||||
@ -58,7 +58,7 @@ jobs:
|
|||||||
mv temp_wheel_dir cu${{ inputs.cu }}_python_deps
|
mv temp_wheel_dir cu${{ inputs.cu }}_python_deps
|
||||||
tar cf cu${{ inputs.cu }}_python_deps.tar cu${{ inputs.cu }}_python_deps
|
tar cf cu${{ inputs.cu }}_python_deps.tar cu${{ inputs.cu }}_python_deps
|
||||||
|
|
||||||
- uses: actions/cache/save@v3
|
- uses: actions/cache/save@v4
|
||||||
with:
|
with:
|
||||||
path: |
|
path: |
|
||||||
cu${{ inputs.cu }}_python_deps.tar
|
cu${{ inputs.cu }}_python_deps.tar
|
||||||
|
|||||||
@ -32,11 +32,11 @@ jobs:
|
|||||||
pull-requests: "read"
|
pull-requests: "read"
|
||||||
runs-on: windows-latest
|
runs-on: windows-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v3
|
- uses: actions/checkout@v4
|
||||||
with:
|
with:
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
persist-credentials: false
|
persist-credentials: false
|
||||||
- uses: actions/setup-python@v4
|
- uses: actions/setup-python@v5
|
||||||
with:
|
with:
|
||||||
python-version: 3.${{ inputs.python_minor }}.${{ inputs.python_patch }}
|
python-version: 3.${{ inputs.python_minor }}.${{ inputs.python_patch }}
|
||||||
- shell: bash
|
- shell: bash
|
||||||
@ -73,7 +73,7 @@ jobs:
|
|||||||
pause" > ./update/update_comfyui_and_python_dependencies.bat
|
pause" > ./update/update_comfyui_and_python_dependencies.bat
|
||||||
cd ..
|
cd ..
|
||||||
|
|
||||||
"C:\Program Files\7-Zip\7z.exe" a -t7z -m0=lzma -mx=8 -mfb=64 -md=32m -ms=on -mf=BCJ2 ComfyUI_windows_portable_nightly_pytorch.7z ComfyUI_windows_portable_nightly_pytorch
|
"C:\Program Files\7-Zip\7z.exe" a -t7z -m0=lzma2 -mx=8 -mfb=64 -md=32m -ms=on -mf=BCJ2 ComfyUI_windows_portable_nightly_pytorch.7z ComfyUI_windows_portable_nightly_pytorch
|
||||||
mv ComfyUI_windows_portable_nightly_pytorch.7z ComfyUI/ComfyUI_windows_portable_nvidia_or_cpu_nightly_pytorch.7z
|
mv ComfyUI_windows_portable_nightly_pytorch.7z ComfyUI/ComfyUI_windows_portable_nvidia_or_cpu_nightly_pytorch.7z
|
||||||
|
|
||||||
cd ComfyUI_windows_portable_nightly_pytorch
|
cd ComfyUI_windows_portable_nightly_pytorch
|
||||||
|
|||||||
@ -32,7 +32,7 @@ jobs:
|
|||||||
pull-requests: "read"
|
pull-requests: "read"
|
||||||
runs-on: windows-latest
|
runs-on: windows-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/cache/restore@v3
|
- uses: actions/cache/restore@v4
|
||||||
id: cache
|
id: cache
|
||||||
with:
|
with:
|
||||||
path: |
|
path: |
|
||||||
@ -48,7 +48,7 @@ jobs:
|
|||||||
pwd
|
pwd
|
||||||
ls
|
ls
|
||||||
|
|
||||||
- uses: actions/checkout@v3
|
- uses: actions/checkout@v4
|
||||||
with:
|
with:
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
persist-credentials: false
|
persist-credentials: false
|
||||||
@ -82,7 +82,7 @@ jobs:
|
|||||||
|
|
||||||
cd ..
|
cd ..
|
||||||
|
|
||||||
"C:\Program Files\7-Zip\7z.exe" a -t7z -m0=lzma -mx=8 -mfb=64 -md=32m -ms=on -mf=BCJ2 ComfyUI_windows_portable.7z ComfyUI_windows_portable
|
"C:\Program Files\7-Zip\7z.exe" a -t7z -m0=lzma2 -mx=8 -mfb=64 -md=32m -ms=on -mf=BCJ2 ComfyUI_windows_portable.7z ComfyUI_windows_portable
|
||||||
mv ComfyUI_windows_portable.7z ComfyUI/new_ComfyUI_windows_portable_nvidia_cu${{ inputs.cu }}_or_cpu.7z
|
mv ComfyUI_windows_portable.7z ComfyUI/new_ComfyUI_windows_portable_nvidia_cu${{ inputs.cu }}_or_cpu.7z
|
||||||
|
|
||||||
cd ComfyUI_windows_portable
|
cd ComfyUI_windows_portable
|
||||||
|
|||||||
3
.gitignore
vendored
3
.gitignore
vendored
@ -9,6 +9,7 @@ __pycache__/
|
|||||||
!custom_nodes/example_node.py.example
|
!custom_nodes/example_node.py.example
|
||||||
extra_model_paths.yaml
|
extra_model_paths.yaml
|
||||||
/.vs
|
/.vs
|
||||||
|
.vscode/
|
||||||
.idea/
|
.idea/
|
||||||
venv/
|
venv/
|
||||||
/web/extensions/*
|
/web/extensions/*
|
||||||
@ -16,3 +17,5 @@ venv/
|
|||||||
!/web/extensions/core/
|
!/web/extensions/core/
|
||||||
/tests-ui/data/object_info.json
|
/tests-ui/data/object_info.json
|
||||||
/user/
|
/user/
|
||||||
|
*.log
|
||||||
|
web_custom_versions/
|
||||||
41
CONTRIBUTING.md
Normal file
41
CONTRIBUTING.md
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
# Contributing to ComfyUI
|
||||||
|
|
||||||
|
Welcome, and thank you for your interest in contributing to ComfyUI!
|
||||||
|
|
||||||
|
There are several ways in which you can contribute, beyond writing code. The goal of this document is to provide a high-level overview of how you can get involved.
|
||||||
|
|
||||||
|
## Asking Questions
|
||||||
|
|
||||||
|
Have a question? Instead of opening an issue, please ask on [Discord](https://comfy.org/discord) or [Matrix](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) channels. Our team and the community will help you.
|
||||||
|
|
||||||
|
## Providing Feedback
|
||||||
|
|
||||||
|
Your comments and feedback are welcome, and the development team is available via a handful of different channels.
|
||||||
|
|
||||||
|
See the `#bug-report`, `#feature-request` and `#feedback` channels on Discord.
|
||||||
|
|
||||||
|
## Reporting Issues
|
||||||
|
|
||||||
|
Have you identified a reproducible problem in ComfyUI? Do you have a feature request? We want to hear about it! Here's how you can report your issue as effectively as possible.
|
||||||
|
|
||||||
|
|
||||||
|
### Look For an Existing Issue
|
||||||
|
|
||||||
|
Before you create a new issue, please do a search in [open issues](https://github.com/comfyanonymous/ComfyUI/issues) to see if the issue or feature request has already been filed.
|
||||||
|
|
||||||
|
If you find your issue already exists, make relevant comments and add your [reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments). Use a reaction in place of a "+1" comment:
|
||||||
|
|
||||||
|
* 👍 - upvote
|
||||||
|
* 👎 - downvote
|
||||||
|
|
||||||
|
If you cannot find an existing issue that describes your bug or feature, create a new issue. We have an issue template in place to organize new issues.
|
||||||
|
|
||||||
|
|
||||||
|
### Creating Pull Requests
|
||||||
|
|
||||||
|
* Please refer to the article on [creating pull requests](https://github.com/comfyanonymous/ComfyUI/wiki/How-to-Contribute-Code) and contributing to this project.
|
||||||
|
|
||||||
|
|
||||||
|
## Thank You
|
||||||
|
|
||||||
|
Your contributions to open source, large or small, make great projects like this possible. Thank you for taking the time to contribute.
|
||||||
23
README.md
23
README.md
@ -11,16 +11,16 @@ This ui will let you design and execute advanced stable diffusion pipelines usin
|
|||||||
|
|
||||||
## Features
|
## Features
|
||||||
- Nodes/graph/flowchart interface to experiment and create complex Stable Diffusion workflows without needing to code anything.
|
- Nodes/graph/flowchart interface to experiment and create complex Stable Diffusion workflows without needing to code anything.
|
||||||
- Fully supports SD1.x, SD2.x, [SDXL](https://comfyanonymous.github.io/ComfyUI_examples/sdxl/), [Stable Video Diffusion](https://comfyanonymous.github.io/ComfyUI_examples/video/) and [Stable Cascade](https://comfyanonymous.github.io/ComfyUI_examples/stable_cascade/)
|
- Fully supports SD1.x, SD2.x, [SDXL](https://comfyanonymous.github.io/ComfyUI_examples/sdxl/), [Stable Video Diffusion](https://comfyanonymous.github.io/ComfyUI_examples/video/), [Stable Cascade](https://comfyanonymous.github.io/ComfyUI_examples/stable_cascade/), [SD3](https://comfyanonymous.github.io/ComfyUI_examples/sd3/) and [Stable Audio](https://comfyanonymous.github.io/ComfyUI_examples/audio/)
|
||||||
- Asynchronous Queue system
|
- Asynchronous Queue system
|
||||||
- Many optimizations: Only re-executes the parts of the workflow that changes between executions.
|
- Many optimizations: Only re-executes the parts of the workflow that changes between executions.
|
||||||
- Command line option: ```--lowvram``` to make it work on GPUs with less than 3GB vram (enabled automatically on GPUs with low vram)
|
- Smart memory management: can automatically run models on GPUs with as low as 1GB vram.
|
||||||
- Works even if you don't have a GPU with: ```--cpu``` (slow)
|
- Works even if you don't have a GPU with: ```--cpu``` (slow)
|
||||||
- Can load ckpt, safetensors and diffusers models/checkpoints. Standalone VAEs and CLIP models.
|
- Can load ckpt, safetensors and diffusers models/checkpoints. Standalone VAEs and CLIP models.
|
||||||
- Embeddings/Textual inversion
|
- Embeddings/Textual inversion
|
||||||
- [Loras (regular, locon and loha)](https://comfyanonymous.github.io/ComfyUI_examples/lora/)
|
- [Loras (regular, locon and loha)](https://comfyanonymous.github.io/ComfyUI_examples/lora/)
|
||||||
- [Hypernetworks](https://comfyanonymous.github.io/ComfyUI_examples/hypernetworks/)
|
- [Hypernetworks](https://comfyanonymous.github.io/ComfyUI_examples/hypernetworks/)
|
||||||
- Loading full workflows (with seeds) from generated PNG files.
|
- Loading full workflows (with seeds) from generated PNG, WebP and FLAC files.
|
||||||
- Saving/Loading workflows as Json files.
|
- Saving/Loading workflows as Json files.
|
||||||
- Nodes interface can be used to create complex workflows like one for [Hires fix](https://comfyanonymous.github.io/ComfyUI_examples/2_pass_txt2img/) or much more advanced ones.
|
- Nodes interface can be used to create complex workflows like one for [Hires fix](https://comfyanonymous.github.io/ComfyUI_examples/2_pass_txt2img/) or much more advanced ones.
|
||||||
- [Area Composition](https://comfyanonymous.github.io/ComfyUI_examples/area_composition/)
|
- [Area Composition](https://comfyanonymous.github.io/ComfyUI_examples/area_composition/)
|
||||||
@ -32,6 +32,8 @@ This ui will let you design and execute advanced stable diffusion pipelines usin
|
|||||||
- [Model Merging](https://comfyanonymous.github.io/ComfyUI_examples/model_merging/)
|
- [Model Merging](https://comfyanonymous.github.io/ComfyUI_examples/model_merging/)
|
||||||
- [LCM models and Loras](https://comfyanonymous.github.io/ComfyUI_examples/lcm/)
|
- [LCM models and Loras](https://comfyanonymous.github.io/ComfyUI_examples/lcm/)
|
||||||
- [SDXL Turbo](https://comfyanonymous.github.io/ComfyUI_examples/sdturbo/)
|
- [SDXL Turbo](https://comfyanonymous.github.io/ComfyUI_examples/sdturbo/)
|
||||||
|
- [AuraFlow](https://comfyanonymous.github.io/ComfyUI_examples/aura_flow/)
|
||||||
|
- [HunyuanDiT](https://comfyanonymous.github.io/ComfyUI_examples/hunyuan_dit/)
|
||||||
- Latent previews with [TAESD](#how-to-show-high-quality-previews)
|
- Latent previews with [TAESD](#how-to-show-high-quality-previews)
|
||||||
- Starts up very fast.
|
- Starts up very fast.
|
||||||
- Works fully offline: will never download anything.
|
- Works fully offline: will never download anything.
|
||||||
@ -76,7 +78,7 @@ Ctrl can also be replaced with Cmd instead for macOS users
|
|||||||
|
|
||||||
There is a portable standalone build for Windows that should work for running on Nvidia GPUs or for running on your CPU only on the [releases page](https://github.com/comfyanonymous/ComfyUI/releases).
|
There is a portable standalone build for Windows that should work for running on Nvidia GPUs or for running on your CPU only on the [releases page](https://github.com/comfyanonymous/ComfyUI/releases).
|
||||||
|
|
||||||
### [Direct link to download](https://github.com/comfyanonymous/ComfyUI/releases/download/latest/ComfyUI_windows_portable_nvidia_cu121_or_cpu.7z)
|
### [Direct link to download](https://github.com/comfyanonymous/ComfyUI/releases/latest/download/ComfyUI_windows_portable_nvidia.7z)
|
||||||
|
|
||||||
Simply download, extract with [7-Zip](https://7-zip.org) and run. Make sure you put your Stable Diffusion checkpoints/models (the huge ckpt/safetensors files) in: ComfyUI\models\checkpoints
|
Simply download, extract with [7-Zip](https://7-zip.org) and run. Make sure you put your Stable Diffusion checkpoints/models (the huge ckpt/safetensors files) in: ComfyUI\models\checkpoints
|
||||||
|
|
||||||
@ -106,7 +108,7 @@ AMD users can install rocm and pytorch with pip if you don't have it already ins
|
|||||||
|
|
||||||
This is the command to install the nightly with ROCm 6.0 which might have some performance improvements:
|
This is the command to install the nightly with ROCm 6.0 which might have some performance improvements:
|
||||||
|
|
||||||
```pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.0```
|
```pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1```
|
||||||
|
|
||||||
### NVIDIA
|
### NVIDIA
|
||||||
|
|
||||||
@ -116,7 +118,7 @@ Nvidia users should install stable pytorch using this command:
|
|||||||
|
|
||||||
This is the command to install pytorch nightly instead which might have performance improvements:
|
This is the command to install pytorch nightly instead which might have performance improvements:
|
||||||
|
|
||||||
```pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121```
|
```pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124```
|
||||||
|
|
||||||
#### Troubleshooting
|
#### Troubleshooting
|
||||||
|
|
||||||
@ -225,12 +227,11 @@ Use `--tls-keyfile key.pem --tls-certfile cert.pem` to enable TLS/SSL, the app w
|
|||||||
|
|
||||||
[Matrix space: #comfyui_space:matrix.org](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) (it's like discord but open source).
|
[Matrix space: #comfyui_space:matrix.org](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org) (it's like discord but open source).
|
||||||
|
|
||||||
|
See also: [https://www.comfy.org/](https://www.comfy.org/)
|
||||||
|
|
||||||
# QA
|
# QA
|
||||||
|
|
||||||
### Why did you make this?
|
### Which GPU should I buy for this?
|
||||||
|
|
||||||
I wanted to learn how Stable Diffusion worked in detail. I also wanted something clean and powerful that would let me experiment with SD without restrictions.
|
[See this page for some recommendations](https://github.com/comfyanonymous/ComfyUI/wiki/Which-GPU-should-I-buy-for-ComfyUI)
|
||||||
|
|
||||||
### Who is this for?
|
|
||||||
|
|
||||||
This is for anyone that wants to make complex workflows with SD or that wants to learn more how SD works. The interface follows closely how SD works and the code should be much more simple to understand than other SD UIs.
|
|
||||||
|
|||||||
188
app/frontend_management.py
Normal file
188
app/frontend_management.py
Normal file
@ -0,0 +1,188 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
import argparse
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import tempfile
|
||||||
|
import zipfile
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from functools import cached_property
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TypedDict
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from typing_extensions import NotRequired
|
||||||
|
from comfy.cli_args import DEFAULT_VERSION_STRING
|
||||||
|
|
||||||
|
|
||||||
|
REQUEST_TIMEOUT = 10 # seconds
|
||||||
|
|
||||||
|
|
||||||
|
class Asset(TypedDict):
|
||||||
|
url: str
|
||||||
|
|
||||||
|
|
||||||
|
class Release(TypedDict):
|
||||||
|
id: int
|
||||||
|
tag_name: str
|
||||||
|
name: str
|
||||||
|
prerelease: bool
|
||||||
|
created_at: str
|
||||||
|
published_at: str
|
||||||
|
body: str
|
||||||
|
assets: NotRequired[list[Asset]]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class FrontEndProvider:
|
||||||
|
owner: str
|
||||||
|
repo: str
|
||||||
|
|
||||||
|
@property
|
||||||
|
def folder_name(self) -> str:
|
||||||
|
return f"{self.owner}_{self.repo}"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def release_url(self) -> str:
|
||||||
|
return f"https://api.github.com/repos/{self.owner}/{self.repo}/releases"
|
||||||
|
|
||||||
|
@cached_property
|
||||||
|
def all_releases(self) -> list[Release]:
|
||||||
|
releases = []
|
||||||
|
api_url = self.release_url
|
||||||
|
while api_url:
|
||||||
|
response = requests.get(api_url, timeout=REQUEST_TIMEOUT)
|
||||||
|
response.raise_for_status() # Raises an HTTPError if the response was an error
|
||||||
|
releases.extend(response.json())
|
||||||
|
# GitHub uses the Link header to provide pagination links. Check if it exists and update api_url accordingly.
|
||||||
|
if "next" in response.links:
|
||||||
|
api_url = response.links["next"]["url"]
|
||||||
|
else:
|
||||||
|
api_url = None
|
||||||
|
return releases
|
||||||
|
|
||||||
|
@cached_property
|
||||||
|
def latest_release(self) -> Release:
|
||||||
|
latest_release_url = f"{self.release_url}/latest"
|
||||||
|
response = requests.get(latest_release_url, timeout=REQUEST_TIMEOUT)
|
||||||
|
response.raise_for_status() # Raises an HTTPError if the response was an error
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def get_release(self, version: str) -> Release:
|
||||||
|
if version == "latest":
|
||||||
|
return self.latest_release
|
||||||
|
else:
|
||||||
|
for release in self.all_releases:
|
||||||
|
if release["tag_name"] in [version, f"v{version}"]:
|
||||||
|
return release
|
||||||
|
raise ValueError(f"Version {version} not found in releases")
|
||||||
|
|
||||||
|
|
||||||
|
def download_release_asset_zip(release: Release, destination_path: str) -> None:
|
||||||
|
"""Download dist.zip from github release."""
|
||||||
|
asset_url = None
|
||||||
|
for asset in release.get("assets", []):
|
||||||
|
if asset["name"] == "dist.zip":
|
||||||
|
asset_url = asset["url"]
|
||||||
|
break
|
||||||
|
|
||||||
|
if not asset_url:
|
||||||
|
raise ValueError("dist.zip not found in the release assets")
|
||||||
|
|
||||||
|
# Use a temporary file to download the zip content
|
||||||
|
with tempfile.TemporaryFile() as tmp_file:
|
||||||
|
headers = {"Accept": "application/octet-stream"}
|
||||||
|
response = requests.get(
|
||||||
|
asset_url, headers=headers, allow_redirects=True, timeout=REQUEST_TIMEOUT
|
||||||
|
)
|
||||||
|
response.raise_for_status() # Ensure we got a successful response
|
||||||
|
|
||||||
|
# Write the content to the temporary file
|
||||||
|
tmp_file.write(response.content)
|
||||||
|
|
||||||
|
# Go back to the beginning of the temporary file
|
||||||
|
tmp_file.seek(0)
|
||||||
|
|
||||||
|
# Extract the zip file content to the destination path
|
||||||
|
with zipfile.ZipFile(tmp_file, "r") as zip_ref:
|
||||||
|
zip_ref.extractall(destination_path)
|
||||||
|
|
||||||
|
|
||||||
|
class FrontendManager:
|
||||||
|
DEFAULT_FRONTEND_PATH = str(Path(__file__).parents[1] / "web")
|
||||||
|
CUSTOM_FRONTENDS_ROOT = str(Path(__file__).parents[1] / "web_custom_versions")
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def parse_version_string(cls, value: str) -> tuple[str, str, str]:
|
||||||
|
"""
|
||||||
|
Args:
|
||||||
|
value (str): The version string to parse.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
tuple[str, str]: A tuple containing provider name and version.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
argparse.ArgumentTypeError: If the version string is invalid.
|
||||||
|
"""
|
||||||
|
VERSION_PATTERN = r"^([a-zA-Z0-9][a-zA-Z0-9-]{0,38})/([a-zA-Z0-9_.-]+)@(v?\d+\.\d+\.\d+|latest)$"
|
||||||
|
match_result = re.match(VERSION_PATTERN, value)
|
||||||
|
if match_result is None:
|
||||||
|
raise argparse.ArgumentTypeError(f"Invalid version string: {value}")
|
||||||
|
|
||||||
|
return match_result.group(1), match_result.group(2), match_result.group(3)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def init_frontend_unsafe(cls, version_string: str) -> str:
|
||||||
|
"""
|
||||||
|
Initializes the frontend for the specified version.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
version_string (str): The version string.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: The path to the initialized frontend.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
Exception: If there is an error during the initialization process.
|
||||||
|
main error source might be request timeout or invalid URL.
|
||||||
|
"""
|
||||||
|
if version_string == DEFAULT_VERSION_STRING:
|
||||||
|
return cls.DEFAULT_FRONTEND_PATH
|
||||||
|
|
||||||
|
repo_owner, repo_name, version = cls.parse_version_string(version_string)
|
||||||
|
provider = FrontEndProvider(repo_owner, repo_name)
|
||||||
|
release = provider.get_release(version)
|
||||||
|
|
||||||
|
semantic_version = release["tag_name"].lstrip("v")
|
||||||
|
web_root = str(
|
||||||
|
Path(cls.CUSTOM_FRONTENDS_ROOT) / provider.folder_name / semantic_version
|
||||||
|
)
|
||||||
|
if not os.path.exists(web_root):
|
||||||
|
os.makedirs(web_root, exist_ok=True)
|
||||||
|
logging.info(
|
||||||
|
"Downloading frontend(%s) version(%s) to (%s)",
|
||||||
|
provider.folder_name,
|
||||||
|
semantic_version,
|
||||||
|
web_root,
|
||||||
|
)
|
||||||
|
logging.debug(release)
|
||||||
|
download_release_asset_zip(release, destination_path=web_root)
|
||||||
|
return web_root
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def init_frontend(cls, version_string: str) -> str:
|
||||||
|
"""
|
||||||
|
Initializes the frontend with the specified version string.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
version_string (str): The version string to initialize the frontend with.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: The path of the initialized frontend.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
return cls.init_frontend_unsafe(version_string)
|
||||||
|
except Exception as e:
|
||||||
|
logging.error("Failed to initialize frontend: %s", e)
|
||||||
|
logging.info("Falling back to the default frontend.")
|
||||||
|
return cls.DEFAULT_FRONTEND_PATH
|
||||||
@ -2,6 +2,8 @@ import json
|
|||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import uuid
|
import uuid
|
||||||
|
import glob
|
||||||
|
import shutil
|
||||||
from aiohttp import web
|
from aiohttp import web
|
||||||
from comfy.cli_args import args
|
from comfy.cli_args import args
|
||||||
from folder_paths import user_directory
|
from folder_paths import user_directory
|
||||||
@ -56,16 +58,16 @@ class UserManager():
|
|||||||
if os.path.commonpath((root_dir, user_root)) != root_dir:
|
if os.path.commonpath((root_dir, user_root)) != root_dir:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
parent = user_root
|
|
||||||
|
|
||||||
if file is not None:
|
if file is not None:
|
||||||
# prevent leaving /{type}/{user}
|
# prevent leaving /{type}/{user}
|
||||||
path = os.path.abspath(os.path.join(user_root, file))
|
path = os.path.abspath(os.path.join(user_root, file))
|
||||||
if os.path.commonpath((user_root, path)) != user_root:
|
if os.path.commonpath((user_root, path)) != user_root:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
parent = os.path.split(path)[0]
|
||||||
|
|
||||||
if create_dir and not os.path.exists(parent):
|
if create_dir and not os.path.exists(parent):
|
||||||
os.mkdir(parent)
|
os.makedirs(parent, exist_ok=True)
|
||||||
|
|
||||||
return path
|
return path
|
||||||
|
|
||||||
@ -108,24 +110,32 @@ class UserManager():
|
|||||||
user_id = self.add_user(username)
|
user_id = self.add_user(username)
|
||||||
return web.json_response(user_id)
|
return web.json_response(user_id)
|
||||||
|
|
||||||
@routes.get("/userdata/{file}")
|
@routes.get("/userdata")
|
||||||
async def getuserdata(request):
|
async def listuserdata(request):
|
||||||
file = request.match_info.get("file", None)
|
directory = request.rel_url.query.get('dir', '')
|
||||||
if not file:
|
if not directory:
|
||||||
return web.Response(status=400)
|
return web.Response(status=400)
|
||||||
|
|
||||||
path = self.get_request_user_filepath(request, file)
|
path = self.get_request_user_filepath(request, directory)
|
||||||
if not path:
|
if not path:
|
||||||
return web.Response(status=403)
|
return web.Response(status=403)
|
||||||
|
|
||||||
if not os.path.exists(path):
|
if not os.path.exists(path):
|
||||||
return web.Response(status=404)
|
return web.Response(status=404)
|
||||||
|
|
||||||
return web.FileResponse(path)
|
recurse = request.rel_url.query.get('recurse', '').lower() == "true"
|
||||||
|
results = glob.glob(os.path.join(
|
||||||
|
glob.escape(path), '**/*'), recursive=recurse)
|
||||||
|
results = [os.path.relpath(x, path) for x in results if os.path.isfile(x)]
|
||||||
|
|
||||||
@routes.post("/userdata/{file}")
|
split_path = request.rel_url.query.get('split', '').lower() == "true"
|
||||||
async def post_userdata(request):
|
if split_path:
|
||||||
file = request.match_info.get("file", None)
|
results = [[x] + x.split(os.sep) for x in results]
|
||||||
|
|
||||||
|
return web.json_response(results)
|
||||||
|
|
||||||
|
def get_user_data_path(request, check_exists = False, param = "file"):
|
||||||
|
file = request.match_info.get(param, None)
|
||||||
if not file:
|
if not file:
|
||||||
return web.Response(status=400)
|
return web.Response(status=400)
|
||||||
|
|
||||||
@ -133,8 +143,63 @@ class UserManager():
|
|||||||
if not path:
|
if not path:
|
||||||
return web.Response(status=403)
|
return web.Response(status=403)
|
||||||
|
|
||||||
|
if check_exists and not os.path.exists(path):
|
||||||
|
return web.Response(status=404)
|
||||||
|
|
||||||
|
return path
|
||||||
|
|
||||||
|
@routes.get("/userdata/{file}")
|
||||||
|
async def getuserdata(request):
|
||||||
|
path = get_user_data_path(request, check_exists=True)
|
||||||
|
if not isinstance(path, str):
|
||||||
|
return path
|
||||||
|
|
||||||
|
return web.FileResponse(path)
|
||||||
|
|
||||||
|
@routes.post("/userdata/{file}")
|
||||||
|
async def post_userdata(request):
|
||||||
|
path = get_user_data_path(request)
|
||||||
|
if not isinstance(path, str):
|
||||||
|
return path
|
||||||
|
|
||||||
|
overwrite = request.query["overwrite"] != "false"
|
||||||
|
if not overwrite and os.path.exists(path):
|
||||||
|
return web.Response(status=409)
|
||||||
|
|
||||||
body = await request.read()
|
body = await request.read()
|
||||||
|
|
||||||
with open(path, "wb") as f:
|
with open(path, "wb") as f:
|
||||||
f.write(body)
|
f.write(body)
|
||||||
|
|
||||||
return web.Response(status=200)
|
resp = os.path.relpath(path, self.get_request_user_filepath(request, None))
|
||||||
|
return web.json_response(resp)
|
||||||
|
|
||||||
|
@routes.delete("/userdata/{file}")
|
||||||
|
async def delete_userdata(request):
|
||||||
|
path = get_user_data_path(request, check_exists=True)
|
||||||
|
if not isinstance(path, str):
|
||||||
|
return path
|
||||||
|
|
||||||
|
os.remove(path)
|
||||||
|
|
||||||
|
return web.Response(status=204)
|
||||||
|
|
||||||
|
@routes.post("/userdata/{file}/move/{dest}")
|
||||||
|
async def move_userdata(request):
|
||||||
|
source = get_user_data_path(request, check_exists=True)
|
||||||
|
if not isinstance(source, str):
|
||||||
|
return source
|
||||||
|
|
||||||
|
dest = get_user_data_path(request, check_exists=False, param="dest")
|
||||||
|
if not isinstance(source, str):
|
||||||
|
return dest
|
||||||
|
|
||||||
|
overwrite = request.query["overwrite"] != "false"
|
||||||
|
if not overwrite and os.path.exists(dest):
|
||||||
|
return web.Response(status=409)
|
||||||
|
|
||||||
|
print(f"moving '{source}' -> '{dest}'")
|
||||||
|
shutil.move(source, dest)
|
||||||
|
|
||||||
|
resp = os.path.relpath(dest, self.get_request_user_filepath(request, None))
|
||||||
|
return web.json_response(resp)
|
||||||
|
|||||||
@ -13,7 +13,47 @@ from ..ldm.modules.diffusionmodules.util import (
|
|||||||
from ..ldm.modules.attention import SpatialTransformer
|
from ..ldm.modules.attention import SpatialTransformer
|
||||||
from ..ldm.modules.diffusionmodules.openaimodel import UNetModel, TimestepEmbedSequential, ResBlock, Downsample
|
from ..ldm.modules.diffusionmodules.openaimodel import UNetModel, TimestepEmbedSequential, ResBlock, Downsample
|
||||||
from ..ldm.util import exists
|
from ..ldm.util import exists
|
||||||
|
from .control_types import UNION_CONTROLNET_TYPES
|
||||||
|
from collections import OrderedDict
|
||||||
import comfy.ops
|
import comfy.ops
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention
|
||||||
|
|
||||||
|
class OptimizedAttention(nn.Module):
|
||||||
|
def __init__(self, c, nhead, dropout=0.0, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.heads = nhead
|
||||||
|
self.c = c
|
||||||
|
|
||||||
|
self.in_proj = operations.Linear(c, c * 3, bias=True, dtype=dtype, device=device)
|
||||||
|
self.out_proj = operations.Linear(c, c, bias=True, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = self.in_proj(x)
|
||||||
|
q, k, v = x.split(self.c, dim=2)
|
||||||
|
out = optimized_attention(q, k, v, self.heads)
|
||||||
|
return self.out_proj(out)
|
||||||
|
|
||||||
|
class QuickGELU(nn.Module):
|
||||||
|
def forward(self, x: torch.Tensor):
|
||||||
|
return x * torch.sigmoid(1.702 * x)
|
||||||
|
|
||||||
|
class ResBlockUnionControlnet(nn.Module):
|
||||||
|
def __init__(self, dim, nhead, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.attn = OptimizedAttention(dim, nhead, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.ln_1 = operations.LayerNorm(dim, dtype=dtype, device=device)
|
||||||
|
self.mlp = nn.Sequential(
|
||||||
|
OrderedDict([("c_fc", operations.Linear(dim, dim * 4, dtype=dtype, device=device)), ("gelu", QuickGELU()),
|
||||||
|
("c_proj", operations.Linear(dim * 4, dim, dtype=dtype, device=device))]))
|
||||||
|
self.ln_2 = operations.LayerNorm(dim, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def attention(self, x: torch.Tensor):
|
||||||
|
return self.attn(x)
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor):
|
||||||
|
x = x + self.attention(self.ln_1(x))
|
||||||
|
x = x + self.mlp(self.ln_2(x))
|
||||||
|
return x
|
||||||
|
|
||||||
class ControlledUnetModel(UNetModel):
|
class ControlledUnetModel(UNetModel):
|
||||||
#implemented in the ldm unet
|
#implemented in the ldm unet
|
||||||
@ -53,6 +93,7 @@ class ControlNet(nn.Module):
|
|||||||
transformer_depth_middle=None,
|
transformer_depth_middle=None,
|
||||||
transformer_depth_output=None,
|
transformer_depth_output=None,
|
||||||
attn_precision=None,
|
attn_precision=None,
|
||||||
|
union_controlnet_num_control_type=None,
|
||||||
device=None,
|
device=None,
|
||||||
operations=comfy.ops.disable_weight_init,
|
operations=comfy.ops.disable_weight_init,
|
||||||
**kwargs,
|
**kwargs,
|
||||||
@ -280,6 +321,65 @@ class ControlNet(nn.Module):
|
|||||||
self.middle_block_out = self.make_zero_conv(ch, operations=operations, dtype=self.dtype, device=device)
|
self.middle_block_out = self.make_zero_conv(ch, operations=operations, dtype=self.dtype, device=device)
|
||||||
self._feature_size += ch
|
self._feature_size += ch
|
||||||
|
|
||||||
|
if union_controlnet_num_control_type is not None:
|
||||||
|
self.num_control_type = union_controlnet_num_control_type
|
||||||
|
num_trans_channel = 320
|
||||||
|
num_trans_head = 8
|
||||||
|
num_trans_layer = 1
|
||||||
|
num_proj_channel = 320
|
||||||
|
# task_scale_factor = num_trans_channel ** 0.5
|
||||||
|
self.task_embedding = nn.Parameter(torch.empty(self.num_control_type, num_trans_channel, dtype=self.dtype, device=device))
|
||||||
|
|
||||||
|
self.transformer_layes = nn.Sequential(*[ResBlockUnionControlnet(num_trans_channel, num_trans_head, dtype=self.dtype, device=device, operations=operations) for _ in range(num_trans_layer)])
|
||||||
|
self.spatial_ch_projs = operations.Linear(num_trans_channel, num_proj_channel, dtype=self.dtype, device=device)
|
||||||
|
#-----------------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
control_add_embed_dim = 256
|
||||||
|
class ControlAddEmbedding(nn.Module):
|
||||||
|
def __init__(self, in_dim, out_dim, num_control_type, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.num_control_type = num_control_type
|
||||||
|
self.in_dim = in_dim
|
||||||
|
self.linear_1 = operations.Linear(in_dim * num_control_type, out_dim, dtype=dtype, device=device)
|
||||||
|
self.linear_2 = operations.Linear(out_dim, out_dim, dtype=dtype, device=device)
|
||||||
|
def forward(self, control_type, dtype, device):
|
||||||
|
c_type = torch.zeros((self.num_control_type,), device=device)
|
||||||
|
c_type[control_type] = 1.0
|
||||||
|
c_type = timestep_embedding(c_type.flatten(), self.in_dim, repeat_only=False).to(dtype).reshape((-1, self.num_control_type * self.in_dim))
|
||||||
|
return self.linear_2(torch.nn.functional.silu(self.linear_1(c_type)))
|
||||||
|
|
||||||
|
self.control_add_embedding = ControlAddEmbedding(control_add_embed_dim, time_embed_dim, self.num_control_type, dtype=self.dtype, device=device, operations=operations)
|
||||||
|
else:
|
||||||
|
self.task_embedding = None
|
||||||
|
self.control_add_embedding = None
|
||||||
|
|
||||||
|
def union_controlnet_merge(self, hint, control_type, emb, context):
|
||||||
|
# Equivalent to: https://github.com/xinsir6/ControlNetPlus/tree/main
|
||||||
|
inputs = []
|
||||||
|
condition_list = []
|
||||||
|
|
||||||
|
for idx in range(min(1, len(control_type))):
|
||||||
|
controlnet_cond = self.input_hint_block(hint[idx], emb, context)
|
||||||
|
feat_seq = torch.mean(controlnet_cond, dim=(2, 3))
|
||||||
|
if idx < len(control_type):
|
||||||
|
feat_seq += self.task_embedding[control_type[idx]].to(dtype=feat_seq.dtype, device=feat_seq.device)
|
||||||
|
|
||||||
|
inputs.append(feat_seq.unsqueeze(1))
|
||||||
|
condition_list.append(controlnet_cond)
|
||||||
|
|
||||||
|
x = torch.cat(inputs, dim=1)
|
||||||
|
x = self.transformer_layes(x)
|
||||||
|
controlnet_cond_fuser = None
|
||||||
|
for idx in range(len(control_type)):
|
||||||
|
alpha = self.spatial_ch_projs(x[:, idx])
|
||||||
|
alpha = alpha.unsqueeze(-1).unsqueeze(-1)
|
||||||
|
o = condition_list[idx] + alpha
|
||||||
|
if controlnet_cond_fuser is None:
|
||||||
|
controlnet_cond_fuser = o
|
||||||
|
else:
|
||||||
|
controlnet_cond_fuser += o
|
||||||
|
return controlnet_cond_fuser
|
||||||
|
|
||||||
def make_zero_conv(self, channels, operations=None, dtype=None, device=None):
|
def make_zero_conv(self, channels, operations=None, dtype=None, device=None):
|
||||||
return TimestepEmbedSequential(operations.conv_nd(self.dims, channels, channels, 1, padding=0, dtype=dtype, device=device))
|
return TimestepEmbedSequential(operations.conv_nd(self.dims, channels, channels, 1, padding=0, dtype=dtype, device=device))
|
||||||
|
|
||||||
@ -287,9 +387,33 @@ class ControlNet(nn.Module):
|
|||||||
t_emb = timestep_embedding(timesteps, self.model_channels, repeat_only=False).to(x.dtype)
|
t_emb = timestep_embedding(timesteps, self.model_channels, repeat_only=False).to(x.dtype)
|
||||||
emb = self.time_embed(t_emb)
|
emb = self.time_embed(t_emb)
|
||||||
|
|
||||||
guided_hint = self.input_hint_block(hint, emb, context)
|
guided_hint = None
|
||||||
|
if self.control_add_embedding is not None: #Union Controlnet
|
||||||
|
control_type = kwargs.get("control_type", [])
|
||||||
|
|
||||||
outs = []
|
if any([c >= self.num_control_type for c in control_type]):
|
||||||
|
max_type = max(control_type)
|
||||||
|
max_type_name = {
|
||||||
|
v: k for k, v in UNION_CONTROLNET_TYPES.items()
|
||||||
|
}[max_type]
|
||||||
|
raise ValueError(
|
||||||
|
f"Control type {max_type_name}({max_type}) is out of range for the number of control types" +
|
||||||
|
f"({self.num_control_type}) supported.\n" +
|
||||||
|
"Please consider using the ProMax ControlNet Union model.\n" +
|
||||||
|
"https://huggingface.co/xinsir/controlnet-union-sdxl-1.0/tree/main"
|
||||||
|
)
|
||||||
|
|
||||||
|
emb += self.control_add_embedding(control_type, emb.dtype, emb.device)
|
||||||
|
if len(control_type) > 0:
|
||||||
|
if len(hint.shape) < 5:
|
||||||
|
hint = hint.unsqueeze(dim=0)
|
||||||
|
guided_hint = self.union_controlnet_merge(hint, control_type, emb, context)
|
||||||
|
|
||||||
|
if guided_hint is None:
|
||||||
|
guided_hint = self.input_hint_block(hint, emb, context)
|
||||||
|
|
||||||
|
out_output = []
|
||||||
|
out_middle = []
|
||||||
|
|
||||||
hs = []
|
hs = []
|
||||||
if self.num_classes is not None:
|
if self.num_classes is not None:
|
||||||
@ -304,10 +428,10 @@ class ControlNet(nn.Module):
|
|||||||
guided_hint = None
|
guided_hint = None
|
||||||
else:
|
else:
|
||||||
h = module(h, emb, context)
|
h = module(h, emb, context)
|
||||||
outs.append(zero_conv(h, emb, context))
|
out_output.append(zero_conv(h, emb, context))
|
||||||
|
|
||||||
h = self.middle_block(h, emb, context)
|
h = self.middle_block(h, emb, context)
|
||||||
outs.append(self.middle_block_out(h, emb, context))
|
out_middle.append(self.middle_block_out(h, emb, context))
|
||||||
|
|
||||||
return outs
|
return {"middle": out_middle, "output": out_output}
|
||||||
|
|
||||||
|
|||||||
10
comfy/cldm/control_types.py
Normal file
10
comfy/cldm/control_types.py
Normal file
@ -0,0 +1,10 @@
|
|||||||
|
UNION_CONTROLNET_TYPES = {
|
||||||
|
"openpose": 0,
|
||||||
|
"depth": 1,
|
||||||
|
"hed/pidi/scribble/ted": 2,
|
||||||
|
"canny/lineart/anime_lineart/mlsd": 3,
|
||||||
|
"normal": 4,
|
||||||
|
"segment": 5,
|
||||||
|
"tile": 6,
|
||||||
|
"repaint": 7,
|
||||||
|
}
|
||||||
77
comfy/cldm/mmdit.py
Normal file
77
comfy/cldm/mmdit.py
Normal file
@ -0,0 +1,77 @@
|
|||||||
|
import torch
|
||||||
|
from typing import Dict, Optional
|
||||||
|
import comfy.ldm.modules.diffusionmodules.mmdit
|
||||||
|
|
||||||
|
class ControlNet(comfy.ldm.modules.diffusionmodules.mmdit.MMDiT):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
num_blocks = None,
|
||||||
|
dtype = None,
|
||||||
|
device = None,
|
||||||
|
operations = None,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
super().__init__(dtype=dtype, device=device, operations=operations, final_layer=False, num_blocks=num_blocks, **kwargs)
|
||||||
|
# controlnet_blocks
|
||||||
|
self.controlnet_blocks = torch.nn.ModuleList([])
|
||||||
|
for _ in range(len(self.joint_blocks)):
|
||||||
|
self.controlnet_blocks.append(operations.Linear(self.hidden_size, self.hidden_size, device=device, dtype=dtype))
|
||||||
|
|
||||||
|
self.pos_embed_input = comfy.ldm.modules.diffusionmodules.mmdit.PatchEmbed(
|
||||||
|
None,
|
||||||
|
self.patch_size,
|
||||||
|
self.in_channels,
|
||||||
|
self.hidden_size,
|
||||||
|
bias=True,
|
||||||
|
strict_img_size=False,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x: torch.Tensor,
|
||||||
|
timesteps: torch.Tensor,
|
||||||
|
y: Optional[torch.Tensor] = None,
|
||||||
|
context: Optional[torch.Tensor] = None,
|
||||||
|
hint = None,
|
||||||
|
) -> torch.Tensor:
|
||||||
|
|
||||||
|
#weird sd3 controlnet specific stuff
|
||||||
|
y = torch.zeros_like(y)
|
||||||
|
|
||||||
|
if self.context_processor is not None:
|
||||||
|
context = self.context_processor(context)
|
||||||
|
|
||||||
|
hw = x.shape[-2:]
|
||||||
|
x = self.x_embedder(x) + self.cropped_pos_embed(hw, device=x.device).to(dtype=x.dtype, device=x.device)
|
||||||
|
x += self.pos_embed_input(hint)
|
||||||
|
|
||||||
|
c = self.t_embedder(timesteps, dtype=x.dtype)
|
||||||
|
if y is not None and self.y_embedder is not None:
|
||||||
|
y = self.y_embedder(y)
|
||||||
|
c = c + y
|
||||||
|
|
||||||
|
if context is not None:
|
||||||
|
context = self.context_embedder(context)
|
||||||
|
|
||||||
|
output = []
|
||||||
|
|
||||||
|
blocks = len(self.joint_blocks)
|
||||||
|
for i in range(blocks):
|
||||||
|
context, x = self.joint_blocks[i](
|
||||||
|
context,
|
||||||
|
x,
|
||||||
|
c=c,
|
||||||
|
use_checkpoint=self.use_checkpoint,
|
||||||
|
)
|
||||||
|
|
||||||
|
out = self.controlnet_blocks[i](x)
|
||||||
|
count = self.depth // blocks
|
||||||
|
if i == blocks - 1:
|
||||||
|
count -= 1
|
||||||
|
for j in range(count):
|
||||||
|
output.append(out)
|
||||||
|
|
||||||
|
return {"output": output}
|
||||||
@ -1,7 +1,10 @@
|
|||||||
import argparse
|
import argparse
|
||||||
import enum
|
import enum
|
||||||
|
import os
|
||||||
|
from typing import Optional
|
||||||
import comfy.options
|
import comfy.options
|
||||||
|
|
||||||
|
|
||||||
class EnumAction(argparse.Action):
|
class EnumAction(argparse.Action):
|
||||||
"""
|
"""
|
||||||
Argparse action for handling Enums
|
Argparse action for handling Enums
|
||||||
@ -75,6 +78,7 @@ fpte_group.add_argument("--fp8_e5m2-text-enc", action="store_true", help="Store
|
|||||||
fpte_group.add_argument("--fp16-text-enc", action="store_true", help="Store text encoder weights in fp16.")
|
fpte_group.add_argument("--fp16-text-enc", action="store_true", help="Store text encoder weights in fp16.")
|
||||||
fpte_group.add_argument("--fp32-text-enc", action="store_true", help="Store text encoder weights in fp32.")
|
fpte_group.add_argument("--fp32-text-enc", action="store_true", help="Store text encoder weights in fp32.")
|
||||||
|
|
||||||
|
parser.add_argument("--force-channels-last", action="store_true", help="Force channels last format when inferencing the models.")
|
||||||
|
|
||||||
parser.add_argument("--directml", type=int, nargs="?", metavar="DIRECTML_DEVICE", const=-1, help="Use torch-directml.")
|
parser.add_argument("--directml", type=int, nargs="?", metavar="DIRECTML_DEVICE", const=-1, help="Use torch-directml.")
|
||||||
|
|
||||||
@ -108,6 +112,7 @@ vram_group.add_argument("--lowvram", action="store_true", help="Split the unet i
|
|||||||
vram_group.add_argument("--novram", action="store_true", help="When lowvram isn't enough.")
|
vram_group.add_argument("--novram", action="store_true", help="When lowvram isn't enough.")
|
||||||
vram_group.add_argument("--cpu", action="store_true", help="To use the CPU for everything (slow).")
|
vram_group.add_argument("--cpu", action="store_true", help="To use the CPU for everything (slow).")
|
||||||
|
|
||||||
|
parser.add_argument("--default-hashing-function", type=str, choices=['md5', 'sha1', 'sha256', 'sha512'], default='sha256', help="Allows you to choose the hash function to use for duplicate filename / contents comparison. Default is sha256.")
|
||||||
|
|
||||||
parser.add_argument("--disable-smart-memory", action="store_true", help="Force ComfyUI to agressively offload to regular ram instead of keeping models in vram when it can.")
|
parser.add_argument("--disable-smart-memory", action="store_true", help="Force ComfyUI to agressively offload to regular ram instead of keeping models in vram when it can.")
|
||||||
parser.add_argument("--deterministic", action="store_true", help="Make pytorch use slower deterministic algorithms when it can. Note that this might not make images deterministic in all cases.")
|
parser.add_argument("--deterministic", action="store_true", help="Make pytorch use slower deterministic algorithms when it can. Note that this might not make images deterministic in all cases.")
|
||||||
@ -117,11 +122,44 @@ parser.add_argument("--quick-test-for-ci", action="store_true", help="Quick test
|
|||||||
parser.add_argument("--windows-standalone-build", action="store_true", help="Windows standalone build: Enable convenient things that most people using the standalone windows build will probably enjoy (like auto opening the page on startup).")
|
parser.add_argument("--windows-standalone-build", action="store_true", help="Windows standalone build: Enable convenient things that most people using the standalone windows build will probably enjoy (like auto opening the page on startup).")
|
||||||
|
|
||||||
parser.add_argument("--disable-metadata", action="store_true", help="Disable saving prompt metadata in files.")
|
parser.add_argument("--disable-metadata", action="store_true", help="Disable saving prompt metadata in files.")
|
||||||
|
parser.add_argument("--disable-all-custom-nodes", action="store_true", help="Disable loading all custom nodes.")
|
||||||
|
|
||||||
parser.add_argument("--multi-user", action="store_true", help="Enables per-user storage.")
|
parser.add_argument("--multi-user", action="store_true", help="Enables per-user storage.")
|
||||||
|
|
||||||
parser.add_argument("--verbose", action="store_true", help="Enables more debug prints.")
|
parser.add_argument("--verbose", action="store_true", help="Enables more debug prints.")
|
||||||
|
|
||||||
|
# The default built-in provider hosted under web/
|
||||||
|
DEFAULT_VERSION_STRING = "comfyanonymous/ComfyUI@latest"
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--front-end-version",
|
||||||
|
type=str,
|
||||||
|
default=DEFAULT_VERSION_STRING,
|
||||||
|
help="""
|
||||||
|
Specifies the version of the frontend to be used. This command needs internet connectivity to query and
|
||||||
|
download available frontend implementations from GitHub releases.
|
||||||
|
|
||||||
|
The version string should be in the format of:
|
||||||
|
[repoOwner]/[repoName]@[version]
|
||||||
|
where version is one of: "latest" or a valid version number (e.g. "1.0.0")
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
|
||||||
|
def is_valid_directory(path: Optional[str]) -> Optional[str]:
|
||||||
|
"""Validate if the given path is a directory."""
|
||||||
|
if path is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if not os.path.isdir(path):
|
||||||
|
raise argparse.ArgumentTypeError(f"{path} is not a valid directory.")
|
||||||
|
return path
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--front-end-root",
|
||||||
|
type=is_valid_directory,
|
||||||
|
default=None,
|
||||||
|
help="The local filesystem path to the directory where the frontend is located. Overrides --front-end-version.",
|
||||||
|
)
|
||||||
|
|
||||||
if comfy.options.args_parsing:
|
if comfy.options.args_parsing:
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|||||||
@ -34,6 +34,7 @@ class ClipVisionModel():
|
|||||||
with open(json_config) as f:
|
with open(json_config) as f:
|
||||||
config = json.load(f)
|
config = json.load(f)
|
||||||
|
|
||||||
|
self.image_size = config.get("image_size", 224)
|
||||||
self.load_device = comfy.model_management.text_encoder_device()
|
self.load_device = comfy.model_management.text_encoder_device()
|
||||||
offload_device = comfy.model_management.text_encoder_offload_device()
|
offload_device = comfy.model_management.text_encoder_offload_device()
|
||||||
self.dtype = comfy.model_management.text_encoder_dtype(self.load_device)
|
self.dtype = comfy.model_management.text_encoder_dtype(self.load_device)
|
||||||
@ -50,7 +51,7 @@ class ClipVisionModel():
|
|||||||
|
|
||||||
def encode_image(self, image):
|
def encode_image(self, image):
|
||||||
comfy.model_management.load_model_gpu(self.patcher)
|
comfy.model_management.load_model_gpu(self.patcher)
|
||||||
pixel_values = clip_preprocess(image.to(self.load_device)).float()
|
pixel_values = clip_preprocess(image.to(self.load_device), size=self.image_size).float()
|
||||||
out = self.model(pixel_values=pixel_values, intermediate_output=-2)
|
out = self.model(pixel_values=pixel_values, intermediate_output=-2)
|
||||||
|
|
||||||
outputs = Output()
|
outputs = Output()
|
||||||
@ -93,7 +94,10 @@ def load_clipvision_from_sd(sd, prefix="", convert_keys=False):
|
|||||||
elif "vision_model.encoder.layers.30.layer_norm1.weight" in sd:
|
elif "vision_model.encoder.layers.30.layer_norm1.weight" in sd:
|
||||||
json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "clip_vision_config_h.json")
|
json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "clip_vision_config_h.json")
|
||||||
elif "vision_model.encoder.layers.22.layer_norm1.weight" in sd:
|
elif "vision_model.encoder.layers.22.layer_norm1.weight" in sd:
|
||||||
json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "clip_vision_config_vitl.json")
|
if sd["vision_model.embeddings.position_embedding.weight"].shape[0] == 577:
|
||||||
|
json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "clip_vision_config_vitl_336.json")
|
||||||
|
else:
|
||||||
|
json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "clip_vision_config_vitl.json")
|
||||||
else:
|
else:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|||||||
18
comfy/clip_vision_config_vitl_336.json
Normal file
18
comfy/clip_vision_config_vitl_336.json
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"dropout": 0.0,
|
||||||
|
"hidden_act": "quick_gelu",
|
||||||
|
"hidden_size": 1024,
|
||||||
|
"image_size": 336,
|
||||||
|
"initializer_factor": 1.0,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 4096,
|
||||||
|
"layer_norm_eps": 1e-5,
|
||||||
|
"model_type": "clip_vision_model",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_channels": 3,
|
||||||
|
"num_hidden_layers": 24,
|
||||||
|
"patch_size": 14,
|
||||||
|
"projection_dim": 768,
|
||||||
|
"torch_dtype": "float32"
|
||||||
|
}
|
||||||
@ -29,7 +29,12 @@ class CONDRegular:
|
|||||||
|
|
||||||
class CONDNoiseShape(CONDRegular):
|
class CONDNoiseShape(CONDRegular):
|
||||||
def process_cond(self, batch_size, device, area, **kwargs):
|
def process_cond(self, batch_size, device, area, **kwargs):
|
||||||
data = self.cond[:,:,area[2]:area[0] + area[2],area[3]:area[1] + area[3]]
|
data = self.cond
|
||||||
|
if area is not None:
|
||||||
|
dims = len(area) // 2
|
||||||
|
for i in range(dims):
|
||||||
|
data = data.narrow(i + 2, area[i + dims], area[i])
|
||||||
|
|
||||||
return self._copy_with(comfy.utils.repeat_to_batch_size(data, batch_size).to(device))
|
return self._copy_with(comfy.utils.repeat_to_batch_size(data, batch_size).to(device))
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -7,10 +7,12 @@ import comfy.model_management
|
|||||||
import comfy.model_detection
|
import comfy.model_detection
|
||||||
import comfy.model_patcher
|
import comfy.model_patcher
|
||||||
import comfy.ops
|
import comfy.ops
|
||||||
|
import comfy.latent_formats
|
||||||
|
|
||||||
import comfy.cldm.cldm
|
import comfy.cldm.cldm
|
||||||
import comfy.t2i_adapter.adapter
|
import comfy.t2i_adapter.adapter
|
||||||
import comfy.ldm.cascade.controlnet
|
import comfy.ldm.cascade.controlnet
|
||||||
|
import comfy.cldm.mmdit
|
||||||
|
|
||||||
|
|
||||||
def broadcast_image_to(tensor, target_batch_size, batched_number):
|
def broadcast_image_to(tensor, target_batch_size, batched_number):
|
||||||
@ -37,20 +39,25 @@ class ControlBase:
|
|||||||
self.cond_hint = None
|
self.cond_hint = None
|
||||||
self.strength = 1.0
|
self.strength = 1.0
|
||||||
self.timestep_percent_range = (0.0, 1.0)
|
self.timestep_percent_range = (0.0, 1.0)
|
||||||
|
self.latent_format = None
|
||||||
|
self.vae = None
|
||||||
self.global_average_pooling = False
|
self.global_average_pooling = False
|
||||||
self.timestep_range = None
|
self.timestep_range = None
|
||||||
self.compression_ratio = 8
|
self.compression_ratio = 8
|
||||||
self.upscale_algorithm = 'nearest-exact'
|
self.upscale_algorithm = 'nearest-exact'
|
||||||
|
self.extra_args = {}
|
||||||
|
|
||||||
if device is None:
|
if device is None:
|
||||||
device = comfy.model_management.get_torch_device()
|
device = comfy.model_management.get_torch_device()
|
||||||
self.device = device
|
self.device = device
|
||||||
self.previous_controlnet = None
|
self.previous_controlnet = None
|
||||||
|
|
||||||
def set_cond_hint(self, cond_hint, strength=1.0, timestep_percent_range=(0.0, 1.0)):
|
def set_cond_hint(self, cond_hint, strength=1.0, timestep_percent_range=(0.0, 1.0), vae=None):
|
||||||
self.cond_hint_original = cond_hint
|
self.cond_hint_original = cond_hint
|
||||||
self.strength = strength
|
self.strength = strength
|
||||||
self.timestep_percent_range = timestep_percent_range
|
self.timestep_percent_range = timestep_percent_range
|
||||||
|
if self.latent_format is not None:
|
||||||
|
self.vae = vae
|
||||||
return self
|
return self
|
||||||
|
|
||||||
def pre_run(self, model, percent_to_timestep_function):
|
def pre_run(self, model, percent_to_timestep_function):
|
||||||
@ -83,43 +90,36 @@ class ControlBase:
|
|||||||
c.global_average_pooling = self.global_average_pooling
|
c.global_average_pooling = self.global_average_pooling
|
||||||
c.compression_ratio = self.compression_ratio
|
c.compression_ratio = self.compression_ratio
|
||||||
c.upscale_algorithm = self.upscale_algorithm
|
c.upscale_algorithm = self.upscale_algorithm
|
||||||
|
c.latent_format = self.latent_format
|
||||||
|
c.extra_args = self.extra_args.copy()
|
||||||
|
c.vae = self.vae
|
||||||
|
|
||||||
def inference_memory_requirements(self, dtype):
|
def inference_memory_requirements(self, dtype):
|
||||||
if self.previous_controlnet is not None:
|
if self.previous_controlnet is not None:
|
||||||
return self.previous_controlnet.inference_memory_requirements(dtype)
|
return self.previous_controlnet.inference_memory_requirements(dtype)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
def control_merge(self, control_input, control_output, control_prev, output_dtype):
|
def control_merge(self, control, control_prev, output_dtype):
|
||||||
out = {'input':[], 'middle':[], 'output': []}
|
out = {'input':[], 'middle':[], 'output': []}
|
||||||
|
|
||||||
if control_input is not None:
|
for key in control:
|
||||||
for i in range(len(control_input)):
|
control_output = control[key]
|
||||||
key = 'input'
|
applied_to = set()
|
||||||
x = control_input[i]
|
|
||||||
if x is not None:
|
|
||||||
x *= self.strength
|
|
||||||
if x.dtype != output_dtype:
|
|
||||||
x = x.to(output_dtype)
|
|
||||||
out[key].insert(0, x)
|
|
||||||
|
|
||||||
if control_output is not None:
|
|
||||||
for i in range(len(control_output)):
|
for i in range(len(control_output)):
|
||||||
if i == (len(control_output) - 1):
|
|
||||||
key = 'middle'
|
|
||||||
index = 0
|
|
||||||
else:
|
|
||||||
key = 'output'
|
|
||||||
index = i
|
|
||||||
x = control_output[i]
|
x = control_output[i]
|
||||||
if x is not None:
|
if x is not None:
|
||||||
if self.global_average_pooling:
|
if self.global_average_pooling:
|
||||||
x = torch.mean(x, dim=(2, 3), keepdim=True).repeat(1, 1, x.shape[2], x.shape[3])
|
x = torch.mean(x, dim=(2, 3), keepdim=True).repeat(1, 1, x.shape[2], x.shape[3])
|
||||||
|
|
||||||
x *= self.strength
|
if x not in applied_to: #memory saving strategy, allow shared tensors and only apply strength to shared tensors once
|
||||||
|
applied_to.add(x)
|
||||||
|
x *= self.strength
|
||||||
|
|
||||||
if x.dtype != output_dtype:
|
if x.dtype != output_dtype:
|
||||||
x = x.to(output_dtype)
|
x = x.to(output_dtype)
|
||||||
|
|
||||||
out[key].append(x)
|
out[key].append(x)
|
||||||
|
|
||||||
if control_prev is not None:
|
if control_prev is not None:
|
||||||
for x in ['input', 'middle', 'output']:
|
for x in ['input', 'middle', 'output']:
|
||||||
o = out[x]
|
o = out[x]
|
||||||
@ -134,20 +134,26 @@ class ControlBase:
|
|||||||
if o[i].shape[0] < prev_val.shape[0]:
|
if o[i].shape[0] < prev_val.shape[0]:
|
||||||
o[i] = prev_val + o[i]
|
o[i] = prev_val + o[i]
|
||||||
else:
|
else:
|
||||||
o[i] += prev_val
|
o[i] = prev_val + o[i] #TODO: change back to inplace add if shared tensors stop being an issue
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
def set_extra_arg(self, argument, value=None):
|
||||||
|
self.extra_args[argument] = value
|
||||||
|
|
||||||
|
|
||||||
class ControlNet(ControlBase):
|
class ControlNet(ControlBase):
|
||||||
def __init__(self, control_model=None, global_average_pooling=False, device=None, load_device=None, manual_cast_dtype=None):
|
def __init__(self, control_model=None, global_average_pooling=False, compression_ratio=8, latent_format=None, device=None, load_device=None, manual_cast_dtype=None):
|
||||||
super().__init__(device)
|
super().__init__(device)
|
||||||
self.control_model = control_model
|
self.control_model = control_model
|
||||||
self.load_device = load_device
|
self.load_device = load_device
|
||||||
if control_model is not None:
|
if control_model is not None:
|
||||||
self.control_model_wrapped = comfy.model_patcher.ModelPatcher(self.control_model, load_device=load_device, offload_device=comfy.model_management.unet_offload_device())
|
self.control_model_wrapped = comfy.model_patcher.ModelPatcher(self.control_model, load_device=load_device, offload_device=comfy.model_management.unet_offload_device())
|
||||||
|
|
||||||
|
self.compression_ratio = compression_ratio
|
||||||
self.global_average_pooling = global_average_pooling
|
self.global_average_pooling = global_average_pooling
|
||||||
self.model_sampling_current = None
|
self.model_sampling_current = None
|
||||||
self.manual_cast_dtype = manual_cast_dtype
|
self.manual_cast_dtype = manual_cast_dtype
|
||||||
|
self.latent_format = latent_format
|
||||||
|
|
||||||
def get_control(self, x_noisy, t, cond, batched_number):
|
def get_control(self, x_noisy, t, cond, batched_number):
|
||||||
control_prev = None
|
control_prev = None
|
||||||
@ -170,7 +176,17 @@ class ControlNet(ControlBase):
|
|||||||
if self.cond_hint is not None:
|
if self.cond_hint is not None:
|
||||||
del self.cond_hint
|
del self.cond_hint
|
||||||
self.cond_hint = None
|
self.cond_hint = None
|
||||||
self.cond_hint = comfy.utils.common_upscale(self.cond_hint_original, x_noisy.shape[3] * self.compression_ratio, x_noisy.shape[2] * self.compression_ratio, self.upscale_algorithm, "center").to(dtype).to(self.device)
|
compression_ratio = self.compression_ratio
|
||||||
|
if self.vae is not None:
|
||||||
|
compression_ratio *= self.vae.downscale_ratio
|
||||||
|
self.cond_hint = comfy.utils.common_upscale(self.cond_hint_original, x_noisy.shape[3] * compression_ratio, x_noisy.shape[2] * compression_ratio, self.upscale_algorithm, "center")
|
||||||
|
if self.vae is not None:
|
||||||
|
loaded_models = comfy.model_management.loaded_models(only_currently_used=True)
|
||||||
|
self.cond_hint = self.vae.encode(self.cond_hint.movedim(1, -1))
|
||||||
|
comfy.model_management.load_models_gpu(loaded_models)
|
||||||
|
if self.latent_format is not None:
|
||||||
|
self.cond_hint = self.latent_format.process_in(self.cond_hint)
|
||||||
|
self.cond_hint = self.cond_hint.to(device=self.device, dtype=dtype)
|
||||||
if x_noisy.shape[0] != self.cond_hint.shape[0]:
|
if x_noisy.shape[0] != self.cond_hint.shape[0]:
|
||||||
self.cond_hint = broadcast_image_to(self.cond_hint, x_noisy.shape[0], batched_number)
|
self.cond_hint = broadcast_image_to(self.cond_hint, x_noisy.shape[0], batched_number)
|
||||||
|
|
||||||
@ -181,8 +197,8 @@ class ControlNet(ControlBase):
|
|||||||
timestep = self.model_sampling_current.timestep(t)
|
timestep = self.model_sampling_current.timestep(t)
|
||||||
x_noisy = self.model_sampling_current.calculate_input(t, x_noisy)
|
x_noisy = self.model_sampling_current.calculate_input(t, x_noisy)
|
||||||
|
|
||||||
control = self.control_model(x=x_noisy.to(dtype), hint=self.cond_hint, timesteps=timestep.float(), context=context.to(dtype), y=y)
|
control = self.control_model(x=x_noisy.to(dtype), hint=self.cond_hint, timesteps=timestep.float(), context=context.to(dtype), y=y, **self.extra_args)
|
||||||
return self.control_merge(None, control, control_prev, output_dtype)
|
return self.control_merge(control, control_prev, output_dtype)
|
||||||
|
|
||||||
def copy(self):
|
def copy(self):
|
||||||
c = ControlNet(None, global_average_pooling=self.global_average_pooling, load_device=self.load_device, manual_cast_dtype=self.manual_cast_dtype)
|
c = ControlNet(None, global_average_pooling=self.global_average_pooling, load_device=self.load_device, manual_cast_dtype=self.manual_cast_dtype)
|
||||||
@ -322,6 +338,39 @@ class ControlLora(ControlNet):
|
|||||||
def inference_memory_requirements(self, dtype):
|
def inference_memory_requirements(self, dtype):
|
||||||
return comfy.utils.calculate_parameters(self.control_weights) * comfy.model_management.dtype_size(dtype) + ControlBase.inference_memory_requirements(self, dtype)
|
return comfy.utils.calculate_parameters(self.control_weights) * comfy.model_management.dtype_size(dtype) + ControlBase.inference_memory_requirements(self, dtype)
|
||||||
|
|
||||||
|
def load_controlnet_mmdit(sd):
|
||||||
|
new_sd = comfy.model_detection.convert_diffusers_mmdit(sd, "")
|
||||||
|
model_config = comfy.model_detection.model_config_from_unet(new_sd, "", True)
|
||||||
|
num_blocks = comfy.model_detection.count_blocks(new_sd, 'joint_blocks.{}.')
|
||||||
|
for k in sd:
|
||||||
|
new_sd[k] = sd[k]
|
||||||
|
|
||||||
|
supported_inference_dtypes = model_config.supported_inference_dtypes
|
||||||
|
|
||||||
|
controlnet_config = model_config.unet_config
|
||||||
|
unet_dtype = comfy.model_management.unet_dtype(supported_dtypes=supported_inference_dtypes)
|
||||||
|
load_device = comfy.model_management.get_torch_device()
|
||||||
|
manual_cast_dtype = comfy.model_management.unet_manual_cast(unet_dtype, load_device)
|
||||||
|
if manual_cast_dtype is not None:
|
||||||
|
operations = comfy.ops.manual_cast
|
||||||
|
else:
|
||||||
|
operations = comfy.ops.disable_weight_init
|
||||||
|
|
||||||
|
control_model = comfy.cldm.mmdit.ControlNet(num_blocks=num_blocks, operations=operations, device=load_device, dtype=unet_dtype, **controlnet_config)
|
||||||
|
missing, unexpected = control_model.load_state_dict(new_sd, strict=False)
|
||||||
|
|
||||||
|
if len(missing) > 0:
|
||||||
|
logging.warning("missing controlnet keys: {}".format(missing))
|
||||||
|
|
||||||
|
if len(unexpected) > 0:
|
||||||
|
logging.debug("unexpected controlnet keys: {}".format(unexpected))
|
||||||
|
|
||||||
|
latent_format = comfy.latent_formats.SD3()
|
||||||
|
latent_format.shift_factor = 0 #SD3 controlnet weirdness
|
||||||
|
control = ControlNet(control_model, compression_ratio=1, latent_format=latent_format, load_device=load_device, manual_cast_dtype=manual_cast_dtype)
|
||||||
|
return control
|
||||||
|
|
||||||
|
|
||||||
def load_controlnet(ckpt_path, model=None):
|
def load_controlnet(ckpt_path, model=None):
|
||||||
controlnet_data = comfy.utils.load_torch_file(ckpt_path, safe_load=True)
|
controlnet_data = comfy.utils.load_torch_file(ckpt_path, safe_load=True)
|
||||||
if "lora_controlnet" in controlnet_data:
|
if "lora_controlnet" in controlnet_data:
|
||||||
@ -370,10 +419,18 @@ def load_controlnet(ckpt_path, model=None):
|
|||||||
if k in controlnet_data:
|
if k in controlnet_data:
|
||||||
new_sd[diffusers_keys[k]] = controlnet_data.pop(k)
|
new_sd[diffusers_keys[k]] = controlnet_data.pop(k)
|
||||||
|
|
||||||
|
if "control_add_embedding.linear_1.bias" in controlnet_data: #Union Controlnet
|
||||||
|
controlnet_config["union_controlnet_num_control_type"] = controlnet_data["task_embedding"].shape[0]
|
||||||
|
for k in list(controlnet_data.keys()):
|
||||||
|
new_k = k.replace('.attn.in_proj_', '.attn.in_proj.')
|
||||||
|
new_sd[new_k] = controlnet_data.pop(k)
|
||||||
|
|
||||||
leftover_keys = controlnet_data.keys()
|
leftover_keys = controlnet_data.keys()
|
||||||
if len(leftover_keys) > 0:
|
if len(leftover_keys) > 0:
|
||||||
logging.warning("leftover keys: {}".format(leftover_keys))
|
logging.warning("leftover keys: {}".format(leftover_keys))
|
||||||
controlnet_data = new_sd
|
controlnet_data = new_sd
|
||||||
|
elif "controlnet_blocks.0.weight" in controlnet_data: #SD3 diffusers format
|
||||||
|
return load_controlnet_mmdit(controlnet_data)
|
||||||
|
|
||||||
pth_key = 'control_model.zero_convs.0.0.weight'
|
pth_key = 'control_model.zero_convs.0.0.weight'
|
||||||
pth = False
|
pth = False
|
||||||
@ -490,12 +547,11 @@ class T2IAdapter(ControlBase):
|
|||||||
self.control_input = self.t2i_model(self.cond_hint.to(x_noisy.dtype))
|
self.control_input = self.t2i_model(self.cond_hint.to(x_noisy.dtype))
|
||||||
self.t2i_model.cpu()
|
self.t2i_model.cpu()
|
||||||
|
|
||||||
control_input = list(map(lambda a: None if a is None else a.clone(), self.control_input))
|
control_input = {}
|
||||||
mid = None
|
for k in self.control_input:
|
||||||
if self.t2i_model.xl == True:
|
control_input[k] = list(map(lambda a: None if a is None else a.clone(), self.control_input[k]))
|
||||||
mid = control_input[-1:]
|
|
||||||
control_input = control_input[:-1]
|
return self.control_merge(control_input, control_prev, x_noisy.dtype)
|
||||||
return self.control_merge(control_input, mid, control_prev, x_noisy.dtype)
|
|
||||||
|
|
||||||
def copy(self):
|
def copy(self):
|
||||||
c = T2IAdapter(self.t2i_model, self.channels_in, self.compression_ratio, self.upscale_algorithm)
|
c = T2IAdapter(self.t2i_model, self.channels_in, self.compression_ratio, self.upscale_algorithm)
|
||||||
|
|||||||
121
comfy/k_diffusion/deis.py
Normal file
121
comfy/k_diffusion/deis.py
Normal file
@ -0,0 +1,121 @@
|
|||||||
|
#Taken from: https://github.com/zju-pi/diff-sampler/blob/main/gits-main/solver_utils.py
|
||||||
|
#under Apache 2 license
|
||||||
|
import torch
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
# A pytorch reimplementation of DEIS (https://github.com/qsh-zh/deis).
|
||||||
|
#############################
|
||||||
|
### Utils for DEIS solver ###
|
||||||
|
#############################
|
||||||
|
#----------------------------------------------------------------------------
|
||||||
|
# Transfer from the input time (sigma) used in EDM to that (t) used in DEIS.
|
||||||
|
|
||||||
|
def edm2t(edm_steps, epsilon_s=1e-3, sigma_min=0.002, sigma_max=80):
|
||||||
|
vp_sigma = lambda beta_d, beta_min: lambda t: (np.e ** (0.5 * beta_d * (t ** 2) + beta_min * t) - 1) ** 0.5
|
||||||
|
vp_sigma_inv = lambda beta_d, beta_min: lambda sigma: ((beta_min ** 2 + 2 * beta_d * (sigma ** 2 + 1).log()).sqrt() - beta_min) / beta_d
|
||||||
|
vp_beta_d = 2 * (np.log(torch.tensor(sigma_min).cpu() ** 2 + 1) / epsilon_s - np.log(torch.tensor(sigma_max).cpu() ** 2 + 1)) / (epsilon_s - 1)
|
||||||
|
vp_beta_min = np.log(torch.tensor(sigma_max).cpu() ** 2 + 1) - 0.5 * vp_beta_d
|
||||||
|
t_steps = vp_sigma_inv(vp_beta_d.clone().detach().cpu(), vp_beta_min.clone().detach().cpu())(edm_steps.clone().detach().cpu())
|
||||||
|
return t_steps, vp_beta_min, vp_beta_d + vp_beta_min
|
||||||
|
|
||||||
|
#----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def cal_poly(prev_t, j, taus):
|
||||||
|
poly = 1
|
||||||
|
for k in range(prev_t.shape[0]):
|
||||||
|
if k == j:
|
||||||
|
continue
|
||||||
|
poly *= (taus - prev_t[k]) / (prev_t[j] - prev_t[k])
|
||||||
|
return poly
|
||||||
|
|
||||||
|
#----------------------------------------------------------------------------
|
||||||
|
# Transfer from t to alpha_t.
|
||||||
|
|
||||||
|
def t2alpha_fn(beta_0, beta_1, t):
|
||||||
|
return torch.exp(-0.5 * t ** 2 * (beta_1 - beta_0) - t * beta_0)
|
||||||
|
|
||||||
|
#----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def cal_intergrand(beta_0, beta_1, taus):
|
||||||
|
with torch.inference_mode(mode=False):
|
||||||
|
taus = taus.clone()
|
||||||
|
beta_0 = beta_0.clone()
|
||||||
|
beta_1 = beta_1.clone()
|
||||||
|
with torch.enable_grad():
|
||||||
|
taus.requires_grad_(True)
|
||||||
|
alpha = t2alpha_fn(beta_0, beta_1, taus)
|
||||||
|
log_alpha = alpha.log()
|
||||||
|
log_alpha.sum().backward()
|
||||||
|
d_log_alpha_dtau = taus.grad
|
||||||
|
integrand = -0.5 * d_log_alpha_dtau / torch.sqrt(alpha * (1 - alpha))
|
||||||
|
return integrand
|
||||||
|
|
||||||
|
#----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def get_deis_coeff_list(t_steps, max_order, N=10000, deis_mode='tab'):
|
||||||
|
"""
|
||||||
|
Get the coefficient list for DEIS sampling.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
t_steps: A pytorch tensor. The time steps for sampling.
|
||||||
|
max_order: A `int`. Maximum order of the solver. 1 <= max_order <= 4
|
||||||
|
N: A `int`. Use how many points to perform the numerical integration when deis_mode=='tab'.
|
||||||
|
deis_mode: A `str`. Select between 'tab' and 'rhoab'. Type of DEIS.
|
||||||
|
Returns:
|
||||||
|
A pytorch tensor. A batch of generated samples or sampling trajectories if return_inters=True.
|
||||||
|
"""
|
||||||
|
if deis_mode == 'tab':
|
||||||
|
t_steps, beta_0, beta_1 = edm2t(t_steps)
|
||||||
|
C = []
|
||||||
|
for i, (t_cur, t_next) in enumerate(zip(t_steps[:-1], t_steps[1:])):
|
||||||
|
order = min(i+1, max_order)
|
||||||
|
if order == 1:
|
||||||
|
C.append([])
|
||||||
|
else:
|
||||||
|
taus = torch.linspace(t_cur, t_next, N) # split the interval for integral appximation
|
||||||
|
dtau = (t_next - t_cur) / N
|
||||||
|
prev_t = t_steps[[i - k for k in range(order)]]
|
||||||
|
coeff_temp = []
|
||||||
|
integrand = cal_intergrand(beta_0, beta_1, taus)
|
||||||
|
for j in range(order):
|
||||||
|
poly = cal_poly(prev_t, j, taus)
|
||||||
|
coeff_temp.append(torch.sum(integrand * poly) * dtau)
|
||||||
|
C.append(coeff_temp)
|
||||||
|
|
||||||
|
elif deis_mode == 'rhoab':
|
||||||
|
# Analytical solution, second order
|
||||||
|
def get_def_intergral_2(a, b, start, end, c):
|
||||||
|
coeff = (end**3 - start**3) / 3 - (end**2 - start**2) * (a + b) / 2 + (end - start) * a * b
|
||||||
|
return coeff / ((c - a) * (c - b))
|
||||||
|
|
||||||
|
# Analytical solution, third order
|
||||||
|
def get_def_intergral_3(a, b, c, start, end, d):
|
||||||
|
coeff = (end**4 - start**4) / 4 - (end**3 - start**3) * (a + b + c) / 3 \
|
||||||
|
+ (end**2 - start**2) * (a*b + a*c + b*c) / 2 - (end - start) * a * b * c
|
||||||
|
return coeff / ((d - a) * (d - b) * (d - c))
|
||||||
|
|
||||||
|
C = []
|
||||||
|
for i, (t_cur, t_next) in enumerate(zip(t_steps[:-1], t_steps[1:])):
|
||||||
|
order = min(i, max_order)
|
||||||
|
if order == 0:
|
||||||
|
C.append([])
|
||||||
|
else:
|
||||||
|
prev_t = t_steps[[i - k for k in range(order+1)]]
|
||||||
|
if order == 1:
|
||||||
|
coeff_cur = ((t_next - prev_t[1])**2 - (t_cur - prev_t[1])**2) / (2 * (t_cur - prev_t[1]))
|
||||||
|
coeff_prev1 = (t_next - t_cur)**2 / (2 * (prev_t[1] - t_cur))
|
||||||
|
coeff_temp = [coeff_cur, coeff_prev1]
|
||||||
|
elif order == 2:
|
||||||
|
coeff_cur = get_def_intergral_2(prev_t[1], prev_t[2], t_cur, t_next, t_cur)
|
||||||
|
coeff_prev1 = get_def_intergral_2(t_cur, prev_t[2], t_cur, t_next, prev_t[1])
|
||||||
|
coeff_prev2 = get_def_intergral_2(t_cur, prev_t[1], t_cur, t_next, prev_t[2])
|
||||||
|
coeff_temp = [coeff_cur, coeff_prev1, coeff_prev2]
|
||||||
|
elif order == 3:
|
||||||
|
coeff_cur = get_def_intergral_3(prev_t[1], prev_t[2], prev_t[3], t_cur, t_next, t_cur)
|
||||||
|
coeff_prev1 = get_def_intergral_3(t_cur, prev_t[2], prev_t[3], t_cur, t_next, prev_t[1])
|
||||||
|
coeff_prev2 = get_def_intergral_3(t_cur, prev_t[1], prev_t[3], t_cur, t_next, prev_t[2])
|
||||||
|
coeff_prev3 = get_def_intergral_3(t_cur, prev_t[1], prev_t[2], t_cur, t_next, prev_t[3])
|
||||||
|
coeff_temp = [coeff_cur, coeff_prev1, coeff_prev2, coeff_prev3]
|
||||||
|
C.append(coeff_temp)
|
||||||
|
return C
|
||||||
|
|
||||||
@ -7,7 +7,8 @@ import torchsde
|
|||||||
from tqdm.auto import trange, tqdm
|
from tqdm.auto import trange, tqdm
|
||||||
|
|
||||||
from . import utils
|
from . import utils
|
||||||
|
from . import deis
|
||||||
|
import comfy.model_patcher
|
||||||
|
|
||||||
def append_zero(x):
|
def append_zero(x):
|
||||||
return torch.cat([x, x.new_zeros([1])])
|
return torch.cat([x, x.new_zeros([1])])
|
||||||
@ -129,8 +130,13 @@ def sample_euler(model, x, sigmas, extra_args=None, callback=None, disable=None,
|
|||||||
extra_args = {} if extra_args is None else extra_args
|
extra_args = {} if extra_args is None else extra_args
|
||||||
s_in = x.new_ones([x.shape[0]])
|
s_in = x.new_ones([x.shape[0]])
|
||||||
for i in trange(len(sigmas) - 1, disable=disable):
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
|
if s_churn > 0:
|
||||||
sigma_hat = sigmas[i] * (gamma + 1)
|
gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
|
||||||
|
sigma_hat = sigmas[i] * (gamma + 1)
|
||||||
|
else:
|
||||||
|
gamma = 0
|
||||||
|
sigma_hat = sigmas[i]
|
||||||
|
|
||||||
if gamma > 0:
|
if gamma > 0:
|
||||||
eps = torch.randn_like(x) * s_noise
|
eps = torch.randn_like(x) * s_noise
|
||||||
x = x + eps * (sigma_hat ** 2 - sigmas[i] ** 2) ** 0.5
|
x = x + eps * (sigma_hat ** 2 - sigmas[i] ** 2) ** 0.5
|
||||||
@ -170,7 +176,13 @@ def sample_heun(model, x, sigmas, extra_args=None, callback=None, disable=None,
|
|||||||
extra_args = {} if extra_args is None else extra_args
|
extra_args = {} if extra_args is None else extra_args
|
||||||
s_in = x.new_ones([x.shape[0]])
|
s_in = x.new_ones([x.shape[0]])
|
||||||
for i in trange(len(sigmas) - 1, disable=disable):
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
|
if s_churn > 0:
|
||||||
|
gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
|
||||||
|
sigma_hat = sigmas[i] * (gamma + 1)
|
||||||
|
else:
|
||||||
|
gamma = 0
|
||||||
|
sigma_hat = sigmas[i]
|
||||||
|
|
||||||
sigma_hat = sigmas[i] * (gamma + 1)
|
sigma_hat = sigmas[i] * (gamma + 1)
|
||||||
if gamma > 0:
|
if gamma > 0:
|
||||||
eps = torch.randn_like(x) * s_noise
|
eps = torch.randn_like(x) * s_noise
|
||||||
@ -199,8 +211,13 @@ def sample_dpm_2(model, x, sigmas, extra_args=None, callback=None, disable=None,
|
|||||||
extra_args = {} if extra_args is None else extra_args
|
extra_args = {} if extra_args is None else extra_args
|
||||||
s_in = x.new_ones([x.shape[0]])
|
s_in = x.new_ones([x.shape[0]])
|
||||||
for i in trange(len(sigmas) - 1, disable=disable):
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
|
if s_churn > 0:
|
||||||
sigma_hat = sigmas[i] * (gamma + 1)
|
gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
|
||||||
|
sigma_hat = sigmas[i] * (gamma + 1)
|
||||||
|
else:
|
||||||
|
gamma = 0
|
||||||
|
sigma_hat = sigmas[i]
|
||||||
|
|
||||||
if gamma > 0:
|
if gamma > 0:
|
||||||
eps = torch.randn_like(x) * s_noise
|
eps = torch.randn_like(x) * s_noise
|
||||||
x = x + eps * (sigma_hat ** 2 - sigmas[i] ** 2) ** 0.5
|
x = x + eps * (sigma_hat ** 2 - sigmas[i] ** 2) ** 0.5
|
||||||
@ -825,3 +842,209 @@ def sample_heunpp2(model, x, sigmas, extra_args=None, callback=None, disable=Non
|
|||||||
d_prime = w1 * d + w2 * d_2 + w3 * d_3
|
d_prime = w1 * d + w2 * d_2 + w3 * d_3
|
||||||
x = x + d_prime * dt
|
x = x + d_prime * dt
|
||||||
return x
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
#From https://github.com/zju-pi/diff-sampler/blob/main/diff-solvers-main/solvers.py
|
||||||
|
#under Apache 2 license
|
||||||
|
def sample_ipndm(model, x, sigmas, extra_args=None, callback=None, disable=None, max_order=4):
|
||||||
|
extra_args = {} if extra_args is None else extra_args
|
||||||
|
s_in = x.new_ones([x.shape[0]])
|
||||||
|
|
||||||
|
x_next = x
|
||||||
|
|
||||||
|
buffer_model = []
|
||||||
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
|
t_cur = sigmas[i]
|
||||||
|
t_next = sigmas[i + 1]
|
||||||
|
|
||||||
|
x_cur = x_next
|
||||||
|
|
||||||
|
denoised = model(x_cur, t_cur * s_in, **extra_args)
|
||||||
|
if callback is not None:
|
||||||
|
callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
|
||||||
|
|
||||||
|
d_cur = (x_cur - denoised) / t_cur
|
||||||
|
|
||||||
|
order = min(max_order, i+1)
|
||||||
|
if order == 1: # First Euler step.
|
||||||
|
x_next = x_cur + (t_next - t_cur) * d_cur
|
||||||
|
elif order == 2: # Use one history point.
|
||||||
|
x_next = x_cur + (t_next - t_cur) * (3 * d_cur - buffer_model[-1]) / 2
|
||||||
|
elif order == 3: # Use two history points.
|
||||||
|
x_next = x_cur + (t_next - t_cur) * (23 * d_cur - 16 * buffer_model[-1] + 5 * buffer_model[-2]) / 12
|
||||||
|
elif order == 4: # Use three history points.
|
||||||
|
x_next = x_cur + (t_next - t_cur) * (55 * d_cur - 59 * buffer_model[-1] + 37 * buffer_model[-2] - 9 * buffer_model[-3]) / 24
|
||||||
|
|
||||||
|
if len(buffer_model) == max_order - 1:
|
||||||
|
for k in range(max_order - 2):
|
||||||
|
buffer_model[k] = buffer_model[k+1]
|
||||||
|
buffer_model[-1] = d_cur
|
||||||
|
else:
|
||||||
|
buffer_model.append(d_cur)
|
||||||
|
|
||||||
|
return x_next
|
||||||
|
|
||||||
|
#From https://github.com/zju-pi/diff-sampler/blob/main/diff-solvers-main/solvers.py
|
||||||
|
#under Apache 2 license
|
||||||
|
def sample_ipndm_v(model, x, sigmas, extra_args=None, callback=None, disable=None, max_order=4):
|
||||||
|
extra_args = {} if extra_args is None else extra_args
|
||||||
|
s_in = x.new_ones([x.shape[0]])
|
||||||
|
|
||||||
|
x_next = x
|
||||||
|
t_steps = sigmas
|
||||||
|
|
||||||
|
buffer_model = []
|
||||||
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
|
t_cur = sigmas[i]
|
||||||
|
t_next = sigmas[i + 1]
|
||||||
|
|
||||||
|
x_cur = x_next
|
||||||
|
|
||||||
|
denoised = model(x_cur, t_cur * s_in, **extra_args)
|
||||||
|
if callback is not None:
|
||||||
|
callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
|
||||||
|
|
||||||
|
d_cur = (x_cur - denoised) / t_cur
|
||||||
|
|
||||||
|
order = min(max_order, i+1)
|
||||||
|
if order == 1: # First Euler step.
|
||||||
|
x_next = x_cur + (t_next - t_cur) * d_cur
|
||||||
|
elif order == 2: # Use one history point.
|
||||||
|
h_n = (t_next - t_cur)
|
||||||
|
h_n_1 = (t_cur - t_steps[i-1])
|
||||||
|
coeff1 = (2 + (h_n / h_n_1)) / 2
|
||||||
|
coeff2 = -(h_n / h_n_1) / 2
|
||||||
|
x_next = x_cur + (t_next - t_cur) * (coeff1 * d_cur + coeff2 * buffer_model[-1])
|
||||||
|
elif order == 3: # Use two history points.
|
||||||
|
h_n = (t_next - t_cur)
|
||||||
|
h_n_1 = (t_cur - t_steps[i-1])
|
||||||
|
h_n_2 = (t_steps[i-1] - t_steps[i-2])
|
||||||
|
temp = (1 - h_n / (3 * (h_n + h_n_1)) * (h_n * (h_n + h_n_1)) / (h_n_1 * (h_n_1 + h_n_2))) / 2
|
||||||
|
coeff1 = (2 + (h_n / h_n_1)) / 2 + temp
|
||||||
|
coeff2 = -(h_n / h_n_1) / 2 - (1 + h_n_1 / h_n_2) * temp
|
||||||
|
coeff3 = temp * h_n_1 / h_n_2
|
||||||
|
x_next = x_cur + (t_next - t_cur) * (coeff1 * d_cur + coeff2 * buffer_model[-1] + coeff3 * buffer_model[-2])
|
||||||
|
elif order == 4: # Use three history points.
|
||||||
|
h_n = (t_next - t_cur)
|
||||||
|
h_n_1 = (t_cur - t_steps[i-1])
|
||||||
|
h_n_2 = (t_steps[i-1] - t_steps[i-2])
|
||||||
|
h_n_3 = (t_steps[i-2] - t_steps[i-3])
|
||||||
|
temp1 = (1 - h_n / (3 * (h_n + h_n_1)) * (h_n * (h_n + h_n_1)) / (h_n_1 * (h_n_1 + h_n_2))) / 2
|
||||||
|
temp2 = ((1 - h_n / (3 * (h_n + h_n_1))) / 2 + (1 - h_n / (2 * (h_n + h_n_1))) * h_n / (6 * (h_n + h_n_1 + h_n_2))) \
|
||||||
|
* (h_n * (h_n + h_n_1) * (h_n + h_n_1 + h_n_2)) / (h_n_1 * (h_n_1 + h_n_2) * (h_n_1 + h_n_2 + h_n_3))
|
||||||
|
coeff1 = (2 + (h_n / h_n_1)) / 2 + temp1 + temp2
|
||||||
|
coeff2 = -(h_n / h_n_1) / 2 - (1 + h_n_1 / h_n_2) * temp1 - (1 + (h_n_1 / h_n_2) + (h_n_1 * (h_n_1 + h_n_2) / (h_n_2 * (h_n_2 + h_n_3)))) * temp2
|
||||||
|
coeff3 = temp1 * h_n_1 / h_n_2 + ((h_n_1 / h_n_2) + (h_n_1 * (h_n_1 + h_n_2) / (h_n_2 * (h_n_2 + h_n_3))) * (1 + h_n_2 / h_n_3)) * temp2
|
||||||
|
coeff4 = -temp2 * (h_n_1 * (h_n_1 + h_n_2) / (h_n_2 * (h_n_2 + h_n_3))) * h_n_1 / h_n_2
|
||||||
|
x_next = x_cur + (t_next - t_cur) * (coeff1 * d_cur + coeff2 * buffer_model[-1] + coeff3 * buffer_model[-2] + coeff4 * buffer_model[-3])
|
||||||
|
|
||||||
|
if len(buffer_model) == max_order - 1:
|
||||||
|
for k in range(max_order - 2):
|
||||||
|
buffer_model[k] = buffer_model[k+1]
|
||||||
|
buffer_model[-1] = d_cur.detach()
|
||||||
|
else:
|
||||||
|
buffer_model.append(d_cur.detach())
|
||||||
|
|
||||||
|
return x_next
|
||||||
|
|
||||||
|
#From https://github.com/zju-pi/diff-sampler/blob/main/diff-solvers-main/solvers.py
|
||||||
|
#under Apache 2 license
|
||||||
|
@torch.no_grad()
|
||||||
|
def sample_deis(model, x, sigmas, extra_args=None, callback=None, disable=None, max_order=3, deis_mode='tab'):
|
||||||
|
extra_args = {} if extra_args is None else extra_args
|
||||||
|
s_in = x.new_ones([x.shape[0]])
|
||||||
|
|
||||||
|
x_next = x
|
||||||
|
t_steps = sigmas
|
||||||
|
|
||||||
|
coeff_list = deis.get_deis_coeff_list(t_steps, max_order, deis_mode=deis_mode)
|
||||||
|
|
||||||
|
buffer_model = []
|
||||||
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
|
t_cur = sigmas[i]
|
||||||
|
t_next = sigmas[i + 1]
|
||||||
|
|
||||||
|
x_cur = x_next
|
||||||
|
|
||||||
|
denoised = model(x_cur, t_cur * s_in, **extra_args)
|
||||||
|
if callback is not None:
|
||||||
|
callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
|
||||||
|
|
||||||
|
d_cur = (x_cur - denoised) / t_cur
|
||||||
|
|
||||||
|
order = min(max_order, i+1)
|
||||||
|
if t_next <= 0:
|
||||||
|
order = 1
|
||||||
|
|
||||||
|
if order == 1: # First Euler step.
|
||||||
|
x_next = x_cur + (t_next - t_cur) * d_cur
|
||||||
|
elif order == 2: # Use one history point.
|
||||||
|
coeff_cur, coeff_prev1 = coeff_list[i]
|
||||||
|
x_next = x_cur + coeff_cur * d_cur + coeff_prev1 * buffer_model[-1]
|
||||||
|
elif order == 3: # Use two history points.
|
||||||
|
coeff_cur, coeff_prev1, coeff_prev2 = coeff_list[i]
|
||||||
|
x_next = x_cur + coeff_cur * d_cur + coeff_prev1 * buffer_model[-1] + coeff_prev2 * buffer_model[-2]
|
||||||
|
elif order == 4: # Use three history points.
|
||||||
|
coeff_cur, coeff_prev1, coeff_prev2, coeff_prev3 = coeff_list[i]
|
||||||
|
x_next = x_cur + coeff_cur * d_cur + coeff_prev1 * buffer_model[-1] + coeff_prev2 * buffer_model[-2] + coeff_prev3 * buffer_model[-3]
|
||||||
|
|
||||||
|
if len(buffer_model) == max_order - 1:
|
||||||
|
for k in range(max_order - 2):
|
||||||
|
buffer_model[k] = buffer_model[k+1]
|
||||||
|
buffer_model[-1] = d_cur.detach()
|
||||||
|
else:
|
||||||
|
buffer_model.append(d_cur.detach())
|
||||||
|
|
||||||
|
return x_next
|
||||||
|
|
||||||
|
@torch.no_grad()
|
||||||
|
def sample_euler_cfg_pp(model, x, sigmas, extra_args=None, callback=None, disable=None):
|
||||||
|
extra_args = {} if extra_args is None else extra_args
|
||||||
|
|
||||||
|
temp = [0]
|
||||||
|
def post_cfg_function(args):
|
||||||
|
temp[0] = args["uncond_denoised"]
|
||||||
|
return args["denoised"]
|
||||||
|
|
||||||
|
model_options = extra_args.get("model_options", {}).copy()
|
||||||
|
extra_args["model_options"] = comfy.model_patcher.set_model_options_post_cfg_function(model_options, post_cfg_function, disable_cfg1_optimization=True)
|
||||||
|
|
||||||
|
s_in = x.new_ones([x.shape[0]])
|
||||||
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
|
sigma_hat = sigmas[i]
|
||||||
|
denoised = model(x, sigma_hat * s_in, **extra_args)
|
||||||
|
d = to_d(x, sigma_hat, temp[0])
|
||||||
|
if callback is not None:
|
||||||
|
callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigma_hat, 'denoised': denoised})
|
||||||
|
dt = sigmas[i + 1] - sigma_hat
|
||||||
|
# Euler method
|
||||||
|
x = denoised + d * sigmas[i + 1]
|
||||||
|
return x
|
||||||
|
|
||||||
|
@torch.no_grad()
|
||||||
|
def sample_euler_ancestral_cfg_pp(model, x, sigmas, extra_args=None, callback=None, disable=None, eta=1., s_noise=1., noise_sampler=None):
|
||||||
|
"""Ancestral sampling with Euler method steps."""
|
||||||
|
extra_args = {} if extra_args is None else extra_args
|
||||||
|
noise_sampler = default_noise_sampler(x) if noise_sampler is None else noise_sampler
|
||||||
|
|
||||||
|
temp = [0]
|
||||||
|
def post_cfg_function(args):
|
||||||
|
temp[0] = args["uncond_denoised"]
|
||||||
|
return args["denoised"]
|
||||||
|
|
||||||
|
model_options = extra_args.get("model_options", {}).copy()
|
||||||
|
extra_args["model_options"] = comfy.model_patcher.set_model_options_post_cfg_function(model_options, post_cfg_function, disable_cfg1_optimization=True)
|
||||||
|
|
||||||
|
s_in = x.new_ones([x.shape[0]])
|
||||||
|
for i in trange(len(sigmas) - 1, disable=disable):
|
||||||
|
denoised = model(x, sigmas[i] * s_in, **extra_args)
|
||||||
|
sigma_down, sigma_up = get_ancestral_step(sigmas[i], sigmas[i + 1], eta=eta)
|
||||||
|
if callback is not None:
|
||||||
|
callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
|
||||||
|
d = to_d(x, sigmas[i], temp[0])
|
||||||
|
# Euler method
|
||||||
|
dt = sigma_down - sigmas[i]
|
||||||
|
x = denoised + d * sigma_down
|
||||||
|
if sigmas[i + 1] > 0:
|
||||||
|
x = x + noise_sampler(sigmas[i], sigmas[i + 1]) * s_noise * sigma_up
|
||||||
|
return x
|
||||||
|
|||||||
@ -2,6 +2,7 @@ import torch
|
|||||||
|
|
||||||
class LatentFormat:
|
class LatentFormat:
|
||||||
scale_factor = 1.0
|
scale_factor = 1.0
|
||||||
|
latent_channels = 4
|
||||||
latent_rgb_factors = None
|
latent_rgb_factors = None
|
||||||
taesd_decoder_name = None
|
taesd_decoder_name = None
|
||||||
|
|
||||||
@ -24,8 +25,9 @@ class SD15(LatentFormat):
|
|||||||
self.taesd_decoder_name = "taesd_decoder"
|
self.taesd_decoder_name = "taesd_decoder"
|
||||||
|
|
||||||
class SDXL(LatentFormat):
|
class SDXL(LatentFormat):
|
||||||
|
scale_factor = 0.13025
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self.scale_factor = 0.13025
|
|
||||||
self.latent_rgb_factors = [
|
self.latent_rgb_factors = [
|
||||||
# R G B
|
# R G B
|
||||||
[ 0.3920, 0.4054, 0.4549],
|
[ 0.3920, 0.4054, 0.4549],
|
||||||
@ -72,6 +74,7 @@ class SD_X4(LatentFormat):
|
|||||||
]
|
]
|
||||||
|
|
||||||
class SC_Prior(LatentFormat):
|
class SC_Prior(LatentFormat):
|
||||||
|
latent_channels = 16
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self.scale_factor = 1.0
|
self.scale_factor = 1.0
|
||||||
self.latent_rgb_factors = [
|
self.latent_rgb_factors = [
|
||||||
@ -102,3 +105,37 @@ class SC_B(LatentFormat):
|
|||||||
[-0.3087, -0.1535, 0.0366],
|
[-0.3087, -0.1535, 0.0366],
|
||||||
[ 0.0290, -0.1574, -0.4078]
|
[ 0.0290, -0.1574, -0.4078]
|
||||||
]
|
]
|
||||||
|
|
||||||
|
class SD3(LatentFormat):
|
||||||
|
latent_channels = 16
|
||||||
|
def __init__(self):
|
||||||
|
self.scale_factor = 1.5305
|
||||||
|
self.shift_factor = 0.0609
|
||||||
|
self.latent_rgb_factors = [
|
||||||
|
[-0.0645, 0.0177, 0.1052],
|
||||||
|
[ 0.0028, 0.0312, 0.0650],
|
||||||
|
[ 0.1848, 0.0762, 0.0360],
|
||||||
|
[ 0.0944, 0.0360, 0.0889],
|
||||||
|
[ 0.0897, 0.0506, -0.0364],
|
||||||
|
[-0.0020, 0.1203, 0.0284],
|
||||||
|
[ 0.0855, 0.0118, 0.0283],
|
||||||
|
[-0.0539, 0.0658, 0.1047],
|
||||||
|
[-0.0057, 0.0116, 0.0700],
|
||||||
|
[-0.0412, 0.0281, -0.0039],
|
||||||
|
[ 0.1106, 0.1171, 0.1220],
|
||||||
|
[-0.0248, 0.0682, -0.0481],
|
||||||
|
[ 0.0815, 0.0846, 0.1207],
|
||||||
|
[-0.0120, -0.0055, -0.0867],
|
||||||
|
[-0.0749, -0.0634, -0.0456],
|
||||||
|
[-0.1418, -0.1457, -0.1259]
|
||||||
|
]
|
||||||
|
self.taesd_decoder_name = "taesd3_decoder"
|
||||||
|
|
||||||
|
def process_in(self, latent):
|
||||||
|
return (latent - self.shift_factor) * self.scale_factor
|
||||||
|
|
||||||
|
def process_out(self, latent):
|
||||||
|
return (latent / self.scale_factor) + self.shift_factor
|
||||||
|
|
||||||
|
class StableAudio1(LatentFormat):
|
||||||
|
latent_channels = 64
|
||||||
|
|||||||
282
comfy/ldm/audio/autoencoder.py
Normal file
282
comfy/ldm/audio/autoencoder.py
Normal file
@ -0,0 +1,282 @@
|
|||||||
|
# code adapted from: https://github.com/Stability-AI/stable-audio-tools
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from torch import nn
|
||||||
|
from typing import Literal, Dict, Any
|
||||||
|
import math
|
||||||
|
import comfy.ops
|
||||||
|
ops = comfy.ops.disable_weight_init
|
||||||
|
|
||||||
|
def vae_sample(mean, scale):
|
||||||
|
stdev = nn.functional.softplus(scale) + 1e-4
|
||||||
|
var = stdev * stdev
|
||||||
|
logvar = torch.log(var)
|
||||||
|
latents = torch.randn_like(mean) * stdev + mean
|
||||||
|
|
||||||
|
kl = (mean * mean + var - logvar - 1).sum(1).mean()
|
||||||
|
|
||||||
|
return latents, kl
|
||||||
|
|
||||||
|
class VAEBottleneck(nn.Module):
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.is_discrete = False
|
||||||
|
|
||||||
|
def encode(self, x, return_info=False, **kwargs):
|
||||||
|
info = {}
|
||||||
|
|
||||||
|
mean, scale = x.chunk(2, dim=1)
|
||||||
|
|
||||||
|
x, kl = vae_sample(mean, scale)
|
||||||
|
|
||||||
|
info["kl"] = kl
|
||||||
|
|
||||||
|
if return_info:
|
||||||
|
return x, info
|
||||||
|
else:
|
||||||
|
return x
|
||||||
|
|
||||||
|
def decode(self, x):
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
def snake_beta(x, alpha, beta):
|
||||||
|
return x + (1.0 / (beta + 0.000000001)) * pow(torch.sin(x * alpha), 2)
|
||||||
|
|
||||||
|
# Adapted from https://github.com/NVIDIA/BigVGAN/blob/main/activations.py under MIT license
|
||||||
|
class SnakeBeta(nn.Module):
|
||||||
|
|
||||||
|
def __init__(self, in_features, alpha=1.0, alpha_trainable=True, alpha_logscale=True):
|
||||||
|
super(SnakeBeta, self).__init__()
|
||||||
|
self.in_features = in_features
|
||||||
|
|
||||||
|
# initialize alpha
|
||||||
|
self.alpha_logscale = alpha_logscale
|
||||||
|
if self.alpha_logscale: # log scale alphas initialized to zeros
|
||||||
|
self.alpha = nn.Parameter(torch.zeros(in_features) * alpha)
|
||||||
|
self.beta = nn.Parameter(torch.zeros(in_features) * alpha)
|
||||||
|
else: # linear scale alphas initialized to ones
|
||||||
|
self.alpha = nn.Parameter(torch.ones(in_features) * alpha)
|
||||||
|
self.beta = nn.Parameter(torch.ones(in_features) * alpha)
|
||||||
|
|
||||||
|
# self.alpha.requires_grad = alpha_trainable
|
||||||
|
# self.beta.requires_grad = alpha_trainable
|
||||||
|
|
||||||
|
self.no_div_by_zero = 0.000000001
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
alpha = self.alpha.unsqueeze(0).unsqueeze(-1).to(x.device) # line up with x to [B, C, T]
|
||||||
|
beta = self.beta.unsqueeze(0).unsqueeze(-1).to(x.device)
|
||||||
|
if self.alpha_logscale:
|
||||||
|
alpha = torch.exp(alpha)
|
||||||
|
beta = torch.exp(beta)
|
||||||
|
x = snake_beta(x, alpha, beta)
|
||||||
|
|
||||||
|
return x
|
||||||
|
|
||||||
|
def WNConv1d(*args, **kwargs):
|
||||||
|
try:
|
||||||
|
return torch.nn.utils.parametrizations.weight_norm(ops.Conv1d(*args, **kwargs))
|
||||||
|
except:
|
||||||
|
return torch.nn.utils.weight_norm(ops.Conv1d(*args, **kwargs)) #support pytorch 2.1 and older
|
||||||
|
|
||||||
|
def WNConvTranspose1d(*args, **kwargs):
|
||||||
|
try:
|
||||||
|
return torch.nn.utils.parametrizations.weight_norm(ops.ConvTranspose1d(*args, **kwargs))
|
||||||
|
except:
|
||||||
|
return torch.nn.utils.weight_norm(ops.ConvTranspose1d(*args, **kwargs)) #support pytorch 2.1 and older
|
||||||
|
|
||||||
|
def get_activation(activation: Literal["elu", "snake", "none"], antialias=False, channels=None) -> nn.Module:
|
||||||
|
if activation == "elu":
|
||||||
|
act = torch.nn.ELU()
|
||||||
|
elif activation == "snake":
|
||||||
|
act = SnakeBeta(channels)
|
||||||
|
elif activation == "none":
|
||||||
|
act = torch.nn.Identity()
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown activation {activation}")
|
||||||
|
|
||||||
|
if antialias:
|
||||||
|
act = Activation1d(act)
|
||||||
|
|
||||||
|
return act
|
||||||
|
|
||||||
|
|
||||||
|
class ResidualUnit(nn.Module):
|
||||||
|
def __init__(self, in_channels, out_channels, dilation, use_snake=False, antialias_activation=False):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.dilation = dilation
|
||||||
|
|
||||||
|
padding = (dilation * (7-1)) // 2
|
||||||
|
|
||||||
|
self.layers = nn.Sequential(
|
||||||
|
get_activation("snake" if use_snake else "elu", antialias=antialias_activation, channels=out_channels),
|
||||||
|
WNConv1d(in_channels=in_channels, out_channels=out_channels,
|
||||||
|
kernel_size=7, dilation=dilation, padding=padding),
|
||||||
|
get_activation("snake" if use_snake else "elu", antialias=antialias_activation, channels=out_channels),
|
||||||
|
WNConv1d(in_channels=out_channels, out_channels=out_channels,
|
||||||
|
kernel_size=1)
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
res = x
|
||||||
|
|
||||||
|
#x = checkpoint(self.layers, x)
|
||||||
|
x = self.layers(x)
|
||||||
|
|
||||||
|
return x + res
|
||||||
|
|
||||||
|
class EncoderBlock(nn.Module):
|
||||||
|
def __init__(self, in_channels, out_channels, stride, use_snake=False, antialias_activation=False):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.layers = nn.Sequential(
|
||||||
|
ResidualUnit(in_channels=in_channels,
|
||||||
|
out_channels=in_channels, dilation=1, use_snake=use_snake),
|
||||||
|
ResidualUnit(in_channels=in_channels,
|
||||||
|
out_channels=in_channels, dilation=3, use_snake=use_snake),
|
||||||
|
ResidualUnit(in_channels=in_channels,
|
||||||
|
out_channels=in_channels, dilation=9, use_snake=use_snake),
|
||||||
|
get_activation("snake" if use_snake else "elu", antialias=antialias_activation, channels=in_channels),
|
||||||
|
WNConv1d(in_channels=in_channels, out_channels=out_channels,
|
||||||
|
kernel_size=2*stride, stride=stride, padding=math.ceil(stride/2)),
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
return self.layers(x)
|
||||||
|
|
||||||
|
class DecoderBlock(nn.Module):
|
||||||
|
def __init__(self, in_channels, out_channels, stride, use_snake=False, antialias_activation=False, use_nearest_upsample=False):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
if use_nearest_upsample:
|
||||||
|
upsample_layer = nn.Sequential(
|
||||||
|
nn.Upsample(scale_factor=stride, mode="nearest"),
|
||||||
|
WNConv1d(in_channels=in_channels,
|
||||||
|
out_channels=out_channels,
|
||||||
|
kernel_size=2*stride,
|
||||||
|
stride=1,
|
||||||
|
bias=False,
|
||||||
|
padding='same')
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
upsample_layer = WNConvTranspose1d(in_channels=in_channels,
|
||||||
|
out_channels=out_channels,
|
||||||
|
kernel_size=2*stride, stride=stride, padding=math.ceil(stride/2))
|
||||||
|
|
||||||
|
self.layers = nn.Sequential(
|
||||||
|
get_activation("snake" if use_snake else "elu", antialias=antialias_activation, channels=in_channels),
|
||||||
|
upsample_layer,
|
||||||
|
ResidualUnit(in_channels=out_channels, out_channels=out_channels,
|
||||||
|
dilation=1, use_snake=use_snake),
|
||||||
|
ResidualUnit(in_channels=out_channels, out_channels=out_channels,
|
||||||
|
dilation=3, use_snake=use_snake),
|
||||||
|
ResidualUnit(in_channels=out_channels, out_channels=out_channels,
|
||||||
|
dilation=9, use_snake=use_snake),
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
return self.layers(x)
|
||||||
|
|
||||||
|
class OobleckEncoder(nn.Module):
|
||||||
|
def __init__(self,
|
||||||
|
in_channels=2,
|
||||||
|
channels=128,
|
||||||
|
latent_dim=32,
|
||||||
|
c_mults = [1, 2, 4, 8],
|
||||||
|
strides = [2, 4, 8, 8],
|
||||||
|
use_snake=False,
|
||||||
|
antialias_activation=False
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
c_mults = [1] + c_mults
|
||||||
|
|
||||||
|
self.depth = len(c_mults)
|
||||||
|
|
||||||
|
layers = [
|
||||||
|
WNConv1d(in_channels=in_channels, out_channels=c_mults[0] * channels, kernel_size=7, padding=3)
|
||||||
|
]
|
||||||
|
|
||||||
|
for i in range(self.depth-1):
|
||||||
|
layers += [EncoderBlock(in_channels=c_mults[i]*channels, out_channels=c_mults[i+1]*channels, stride=strides[i], use_snake=use_snake)]
|
||||||
|
|
||||||
|
layers += [
|
||||||
|
get_activation("snake" if use_snake else "elu", antialias=antialias_activation, channels=c_mults[-1] * channels),
|
||||||
|
WNConv1d(in_channels=c_mults[-1]*channels, out_channels=latent_dim, kernel_size=3, padding=1)
|
||||||
|
]
|
||||||
|
|
||||||
|
self.layers = nn.Sequential(*layers)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
return self.layers(x)
|
||||||
|
|
||||||
|
|
||||||
|
class OobleckDecoder(nn.Module):
|
||||||
|
def __init__(self,
|
||||||
|
out_channels=2,
|
||||||
|
channels=128,
|
||||||
|
latent_dim=32,
|
||||||
|
c_mults = [1, 2, 4, 8],
|
||||||
|
strides = [2, 4, 8, 8],
|
||||||
|
use_snake=False,
|
||||||
|
antialias_activation=False,
|
||||||
|
use_nearest_upsample=False,
|
||||||
|
final_tanh=True):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
c_mults = [1] + c_mults
|
||||||
|
|
||||||
|
self.depth = len(c_mults)
|
||||||
|
|
||||||
|
layers = [
|
||||||
|
WNConv1d(in_channels=latent_dim, out_channels=c_mults[-1]*channels, kernel_size=7, padding=3),
|
||||||
|
]
|
||||||
|
|
||||||
|
for i in range(self.depth-1, 0, -1):
|
||||||
|
layers += [DecoderBlock(
|
||||||
|
in_channels=c_mults[i]*channels,
|
||||||
|
out_channels=c_mults[i-1]*channels,
|
||||||
|
stride=strides[i-1],
|
||||||
|
use_snake=use_snake,
|
||||||
|
antialias_activation=antialias_activation,
|
||||||
|
use_nearest_upsample=use_nearest_upsample
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
layers += [
|
||||||
|
get_activation("snake" if use_snake else "elu", antialias=antialias_activation, channels=c_mults[0] * channels),
|
||||||
|
WNConv1d(in_channels=c_mults[0] * channels, out_channels=out_channels, kernel_size=7, padding=3, bias=False),
|
||||||
|
nn.Tanh() if final_tanh else nn.Identity()
|
||||||
|
]
|
||||||
|
|
||||||
|
self.layers = nn.Sequential(*layers)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
return self.layers(x)
|
||||||
|
|
||||||
|
|
||||||
|
class AudioOobleckVAE(nn.Module):
|
||||||
|
def __init__(self,
|
||||||
|
in_channels=2,
|
||||||
|
channels=128,
|
||||||
|
latent_dim=64,
|
||||||
|
c_mults = [1, 2, 4, 8, 16],
|
||||||
|
strides = [2, 4, 4, 8, 8],
|
||||||
|
use_snake=True,
|
||||||
|
antialias_activation=False,
|
||||||
|
use_nearest_upsample=False,
|
||||||
|
final_tanh=False):
|
||||||
|
super().__init__()
|
||||||
|
self.encoder = OobleckEncoder(in_channels, channels, latent_dim * 2, c_mults, strides, use_snake, antialias_activation)
|
||||||
|
self.decoder = OobleckDecoder(in_channels, channels, latent_dim, c_mults, strides, use_snake, antialias_activation,
|
||||||
|
use_nearest_upsample=use_nearest_upsample, final_tanh=final_tanh)
|
||||||
|
self.bottleneck = VAEBottleneck()
|
||||||
|
|
||||||
|
def encode(self, x):
|
||||||
|
return self.bottleneck.encode(self.encoder(x))
|
||||||
|
|
||||||
|
def decode(self, x):
|
||||||
|
return self.decoder(self.bottleneck.decode(x))
|
||||||
|
|
||||||
888
comfy/ldm/audio/dit.py
Normal file
888
comfy/ldm/audio/dit.py
Normal file
@ -0,0 +1,888 @@
|
|||||||
|
# code adapted from: https://github.com/Stability-AI/stable-audio-tools
|
||||||
|
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention
|
||||||
|
import typing as tp
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from einops import rearrange
|
||||||
|
from torch import nn
|
||||||
|
from torch.nn import functional as F
|
||||||
|
import math
|
||||||
|
|
||||||
|
class FourierFeatures(nn.Module):
|
||||||
|
def __init__(self, in_features, out_features, std=1., dtype=None, device=None):
|
||||||
|
super().__init__()
|
||||||
|
assert out_features % 2 == 0
|
||||||
|
self.weight = nn.Parameter(torch.empty(
|
||||||
|
[out_features // 2, in_features], dtype=dtype, device=device))
|
||||||
|
|
||||||
|
def forward(self, input):
|
||||||
|
f = 2 * math.pi * input @ self.weight.T.to(dtype=input.dtype, device=input.device)
|
||||||
|
return torch.cat([f.cos(), f.sin()], dim=-1)
|
||||||
|
|
||||||
|
# norms
|
||||||
|
class LayerNorm(nn.Module):
|
||||||
|
def __init__(self, dim, bias=False, fix_scale=False, dtype=None, device=None):
|
||||||
|
"""
|
||||||
|
bias-less layernorm has been shown to be more stable. most newer models have moved towards rmsnorm, also bias-less
|
||||||
|
"""
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.gamma = nn.Parameter(torch.empty(dim, dtype=dtype, device=device))
|
||||||
|
|
||||||
|
if bias:
|
||||||
|
self.beta = nn.Parameter(torch.empty(dim, dtype=dtype, device=device))
|
||||||
|
else:
|
||||||
|
self.beta = None
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
beta = self.beta
|
||||||
|
if self.beta is not None:
|
||||||
|
beta = beta.to(dtype=x.dtype, device=x.device)
|
||||||
|
return F.layer_norm(x, x.shape[-1:], weight=self.gamma.to(dtype=x.dtype, device=x.device), bias=beta)
|
||||||
|
|
||||||
|
class GLU(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim_in,
|
||||||
|
dim_out,
|
||||||
|
activation,
|
||||||
|
use_conv = False,
|
||||||
|
conv_kernel_size = 3,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.act = activation
|
||||||
|
self.proj = operations.Linear(dim_in, dim_out * 2, dtype=dtype, device=device) if not use_conv else operations.Conv1d(dim_in, dim_out * 2, conv_kernel_size, padding = (conv_kernel_size // 2), dtype=dtype, device=device)
|
||||||
|
self.use_conv = use_conv
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
if self.use_conv:
|
||||||
|
x = rearrange(x, 'b n d -> b d n')
|
||||||
|
x = self.proj(x)
|
||||||
|
x = rearrange(x, 'b d n -> b n d')
|
||||||
|
else:
|
||||||
|
x = self.proj(x)
|
||||||
|
|
||||||
|
x, gate = x.chunk(2, dim = -1)
|
||||||
|
return x * self.act(gate)
|
||||||
|
|
||||||
|
class AbsolutePositionalEmbedding(nn.Module):
|
||||||
|
def __init__(self, dim, max_seq_len):
|
||||||
|
super().__init__()
|
||||||
|
self.scale = dim ** -0.5
|
||||||
|
self.max_seq_len = max_seq_len
|
||||||
|
self.emb = nn.Embedding(max_seq_len, dim)
|
||||||
|
|
||||||
|
def forward(self, x, pos = None, seq_start_pos = None):
|
||||||
|
seq_len, device = x.shape[1], x.device
|
||||||
|
assert seq_len <= self.max_seq_len, f'you are passing in a sequence length of {seq_len} but your absolute positional embedding has a max sequence length of {self.max_seq_len}'
|
||||||
|
|
||||||
|
if pos is None:
|
||||||
|
pos = torch.arange(seq_len, device = device)
|
||||||
|
|
||||||
|
if seq_start_pos is not None:
|
||||||
|
pos = (pos - seq_start_pos[..., None]).clamp(min = 0)
|
||||||
|
|
||||||
|
pos_emb = self.emb(pos)
|
||||||
|
pos_emb = pos_emb * self.scale
|
||||||
|
return pos_emb
|
||||||
|
|
||||||
|
class ScaledSinusoidalEmbedding(nn.Module):
|
||||||
|
def __init__(self, dim, theta = 10000):
|
||||||
|
super().__init__()
|
||||||
|
assert (dim % 2) == 0, 'dimension must be divisible by 2'
|
||||||
|
self.scale = nn.Parameter(torch.ones(1) * dim ** -0.5)
|
||||||
|
|
||||||
|
half_dim = dim // 2
|
||||||
|
freq_seq = torch.arange(half_dim).float() / half_dim
|
||||||
|
inv_freq = theta ** -freq_seq
|
||||||
|
self.register_buffer('inv_freq', inv_freq, persistent = False)
|
||||||
|
|
||||||
|
def forward(self, x, pos = None, seq_start_pos = None):
|
||||||
|
seq_len, device = x.shape[1], x.device
|
||||||
|
|
||||||
|
if pos is None:
|
||||||
|
pos = torch.arange(seq_len, device = device)
|
||||||
|
|
||||||
|
if seq_start_pos is not None:
|
||||||
|
pos = pos - seq_start_pos[..., None]
|
||||||
|
|
||||||
|
emb = torch.einsum('i, j -> i j', pos, self.inv_freq)
|
||||||
|
emb = torch.cat((emb.sin(), emb.cos()), dim = -1)
|
||||||
|
return emb * self.scale
|
||||||
|
|
||||||
|
class RotaryEmbedding(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim,
|
||||||
|
use_xpos = False,
|
||||||
|
scale_base = 512,
|
||||||
|
interpolation_factor = 1.,
|
||||||
|
base = 10000,
|
||||||
|
base_rescale_factor = 1.
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
# proposed by reddit user bloc97, to rescale rotary embeddings to longer sequence length without fine-tuning
|
||||||
|
# has some connection to NTK literature
|
||||||
|
# https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/
|
||||||
|
base *= base_rescale_factor ** (dim / (dim - 2))
|
||||||
|
|
||||||
|
inv_freq = 1. / (base ** (torch.arange(0, dim, 2).float() / dim))
|
||||||
|
self.register_buffer('inv_freq', inv_freq)
|
||||||
|
|
||||||
|
assert interpolation_factor >= 1.
|
||||||
|
self.interpolation_factor = interpolation_factor
|
||||||
|
|
||||||
|
if not use_xpos:
|
||||||
|
self.register_buffer('scale', None)
|
||||||
|
return
|
||||||
|
|
||||||
|
scale = (torch.arange(0, dim, 2) + 0.4 * dim) / (1.4 * dim)
|
||||||
|
|
||||||
|
self.scale_base = scale_base
|
||||||
|
self.register_buffer('scale', scale)
|
||||||
|
|
||||||
|
def forward_from_seq_len(self, seq_len, device, dtype):
|
||||||
|
# device = self.inv_freq.device
|
||||||
|
|
||||||
|
t = torch.arange(seq_len, device=device, dtype=dtype)
|
||||||
|
return self.forward(t)
|
||||||
|
|
||||||
|
def forward(self, t):
|
||||||
|
# device = self.inv_freq.device
|
||||||
|
device = t.device
|
||||||
|
dtype = t.dtype
|
||||||
|
|
||||||
|
# t = t.to(torch.float32)
|
||||||
|
|
||||||
|
t = t / self.interpolation_factor
|
||||||
|
|
||||||
|
freqs = torch.einsum('i , j -> i j', t, self.inv_freq.to(dtype=dtype, device=device))
|
||||||
|
freqs = torch.cat((freqs, freqs), dim = -1)
|
||||||
|
|
||||||
|
if self.scale is None:
|
||||||
|
return freqs, 1.
|
||||||
|
|
||||||
|
power = (torch.arange(seq_len, device = device) - (seq_len // 2)) / self.scale_base
|
||||||
|
scale = self.scale.to(dtype=dtype, device=device) ** rearrange(power, 'n -> n 1')
|
||||||
|
scale = torch.cat((scale, scale), dim = -1)
|
||||||
|
|
||||||
|
return freqs, scale
|
||||||
|
|
||||||
|
def rotate_half(x):
|
||||||
|
x = rearrange(x, '... (j d) -> ... j d', j = 2)
|
||||||
|
x1, x2 = x.unbind(dim = -2)
|
||||||
|
return torch.cat((-x2, x1), dim = -1)
|
||||||
|
|
||||||
|
def apply_rotary_pos_emb(t, freqs, scale = 1):
|
||||||
|
out_dtype = t.dtype
|
||||||
|
|
||||||
|
# cast to float32 if necessary for numerical stability
|
||||||
|
dtype = t.dtype #reduce(torch.promote_types, (t.dtype, freqs.dtype, torch.float32))
|
||||||
|
rot_dim, seq_len = freqs.shape[-1], t.shape[-2]
|
||||||
|
freqs, t = freqs.to(dtype), t.to(dtype)
|
||||||
|
freqs = freqs[-seq_len:, :]
|
||||||
|
|
||||||
|
if t.ndim == 4 and freqs.ndim == 3:
|
||||||
|
freqs = rearrange(freqs, 'b n d -> b 1 n d')
|
||||||
|
|
||||||
|
# partial rotary embeddings, Wang et al. GPT-J
|
||||||
|
t, t_unrotated = t[..., :rot_dim], t[..., rot_dim:]
|
||||||
|
t = (t * freqs.cos() * scale) + (rotate_half(t) * freqs.sin() * scale)
|
||||||
|
|
||||||
|
t, t_unrotated = t.to(out_dtype), t_unrotated.to(out_dtype)
|
||||||
|
|
||||||
|
return torch.cat((t, t_unrotated), dim = -1)
|
||||||
|
|
||||||
|
class FeedForward(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim,
|
||||||
|
dim_out = None,
|
||||||
|
mult = 4,
|
||||||
|
no_bias = False,
|
||||||
|
glu = True,
|
||||||
|
use_conv = False,
|
||||||
|
conv_kernel_size = 3,
|
||||||
|
zero_init_output = True,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
inner_dim = int(dim * mult)
|
||||||
|
|
||||||
|
# Default to SwiGLU
|
||||||
|
|
||||||
|
activation = nn.SiLU()
|
||||||
|
|
||||||
|
dim_out = dim if dim_out is None else dim_out
|
||||||
|
|
||||||
|
if glu:
|
||||||
|
linear_in = GLU(dim, inner_dim, activation, dtype=dtype, device=device, operations=operations)
|
||||||
|
else:
|
||||||
|
linear_in = nn.Sequential(
|
||||||
|
Rearrange('b n d -> b d n') if use_conv else nn.Identity(),
|
||||||
|
operations.Linear(dim, inner_dim, bias = not no_bias, dtype=dtype, device=device) if not use_conv else operations.Conv1d(dim, inner_dim, conv_kernel_size, padding = (conv_kernel_size // 2), bias = not no_bias, dtype=dtype, device=device),
|
||||||
|
Rearrange('b n d -> b d n') if use_conv else nn.Identity(),
|
||||||
|
activation
|
||||||
|
)
|
||||||
|
|
||||||
|
linear_out = operations.Linear(inner_dim, dim_out, bias = not no_bias, dtype=dtype, device=device) if not use_conv else operations.Conv1d(inner_dim, dim_out, conv_kernel_size, padding = (conv_kernel_size // 2), bias = not no_bias, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
# # init last linear layer to 0
|
||||||
|
# if zero_init_output:
|
||||||
|
# nn.init.zeros_(linear_out.weight)
|
||||||
|
# if not no_bias:
|
||||||
|
# nn.init.zeros_(linear_out.bias)
|
||||||
|
|
||||||
|
|
||||||
|
self.ff = nn.Sequential(
|
||||||
|
linear_in,
|
||||||
|
Rearrange('b d n -> b n d') if use_conv else nn.Identity(),
|
||||||
|
linear_out,
|
||||||
|
Rearrange('b n d -> b d n') if use_conv else nn.Identity(),
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
return self.ff(x)
|
||||||
|
|
||||||
|
class Attention(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim,
|
||||||
|
dim_heads = 64,
|
||||||
|
dim_context = None,
|
||||||
|
causal = False,
|
||||||
|
zero_init_output=True,
|
||||||
|
qk_norm = False,
|
||||||
|
natten_kernel_size = None,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.dim = dim
|
||||||
|
self.dim_heads = dim_heads
|
||||||
|
self.causal = causal
|
||||||
|
|
||||||
|
dim_kv = dim_context if dim_context is not None else dim
|
||||||
|
|
||||||
|
self.num_heads = dim // dim_heads
|
||||||
|
self.kv_heads = dim_kv // dim_heads
|
||||||
|
|
||||||
|
if dim_context is not None:
|
||||||
|
self.to_q = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.to_kv = operations.Linear(dim_kv, dim_kv * 2, bias=False, dtype=dtype, device=device)
|
||||||
|
else:
|
||||||
|
self.to_qkv = operations.Linear(dim, dim * 3, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.to_out = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
# if zero_init_output:
|
||||||
|
# nn.init.zeros_(self.to_out.weight)
|
||||||
|
|
||||||
|
self.qk_norm = qk_norm
|
||||||
|
|
||||||
|
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
context = None,
|
||||||
|
mask = None,
|
||||||
|
context_mask = None,
|
||||||
|
rotary_pos_emb = None,
|
||||||
|
causal = None
|
||||||
|
):
|
||||||
|
h, kv_h, has_context = self.num_heads, self.kv_heads, context is not None
|
||||||
|
|
||||||
|
kv_input = context if has_context else x
|
||||||
|
|
||||||
|
if hasattr(self, 'to_q'):
|
||||||
|
# Use separate linear projections for q and k/v
|
||||||
|
q = self.to_q(x)
|
||||||
|
q = rearrange(q, 'b n (h d) -> b h n d', h = h)
|
||||||
|
|
||||||
|
k, v = self.to_kv(kv_input).chunk(2, dim=-1)
|
||||||
|
|
||||||
|
k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = kv_h), (k, v))
|
||||||
|
else:
|
||||||
|
# Use fused linear projection
|
||||||
|
q, k, v = self.to_qkv(x).chunk(3, dim=-1)
|
||||||
|
q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = h), (q, k, v))
|
||||||
|
|
||||||
|
# Normalize q and k for cosine sim attention
|
||||||
|
if self.qk_norm:
|
||||||
|
q = F.normalize(q, dim=-1)
|
||||||
|
k = F.normalize(k, dim=-1)
|
||||||
|
|
||||||
|
if rotary_pos_emb is not None and not has_context:
|
||||||
|
freqs, _ = rotary_pos_emb
|
||||||
|
|
||||||
|
q_dtype = q.dtype
|
||||||
|
k_dtype = k.dtype
|
||||||
|
|
||||||
|
q = q.to(torch.float32)
|
||||||
|
k = k.to(torch.float32)
|
||||||
|
freqs = freqs.to(torch.float32)
|
||||||
|
|
||||||
|
q = apply_rotary_pos_emb(q, freqs)
|
||||||
|
k = apply_rotary_pos_emb(k, freqs)
|
||||||
|
|
||||||
|
q = q.to(q_dtype)
|
||||||
|
k = k.to(k_dtype)
|
||||||
|
|
||||||
|
input_mask = context_mask
|
||||||
|
|
||||||
|
if input_mask is None and not has_context:
|
||||||
|
input_mask = mask
|
||||||
|
|
||||||
|
# determine masking
|
||||||
|
masks = []
|
||||||
|
final_attn_mask = None # The mask that will be applied to the attention matrix, taking all masks into account
|
||||||
|
|
||||||
|
if input_mask is not None:
|
||||||
|
input_mask = rearrange(input_mask, 'b j -> b 1 1 j')
|
||||||
|
masks.append(~input_mask)
|
||||||
|
|
||||||
|
# Other masks will be added here later
|
||||||
|
|
||||||
|
if len(masks) > 0:
|
||||||
|
final_attn_mask = ~or_reduce(masks)
|
||||||
|
|
||||||
|
n, device = q.shape[-2], q.device
|
||||||
|
|
||||||
|
causal = self.causal if causal is None else causal
|
||||||
|
|
||||||
|
if n == 1 and causal:
|
||||||
|
causal = False
|
||||||
|
|
||||||
|
if h != kv_h:
|
||||||
|
# Repeat interleave kv_heads to match q_heads
|
||||||
|
heads_per_kv_head = h // kv_h
|
||||||
|
k, v = map(lambda t: t.repeat_interleave(heads_per_kv_head, dim = 1), (k, v))
|
||||||
|
|
||||||
|
out = optimized_attention(q, k, v, h, skip_reshape=True)
|
||||||
|
out = self.to_out(out)
|
||||||
|
|
||||||
|
if mask is not None:
|
||||||
|
mask = rearrange(mask, 'b n -> b n 1')
|
||||||
|
out = out.masked_fill(~mask, 0.)
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
class ConformerModule(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim,
|
||||||
|
norm_kwargs = {},
|
||||||
|
):
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.dim = dim
|
||||||
|
|
||||||
|
self.in_norm = LayerNorm(dim, **norm_kwargs)
|
||||||
|
self.pointwise_conv = nn.Conv1d(dim, dim, kernel_size=1, bias=False)
|
||||||
|
self.glu = GLU(dim, dim, nn.SiLU())
|
||||||
|
self.depthwise_conv = nn.Conv1d(dim, dim, kernel_size=17, groups=dim, padding=8, bias=False)
|
||||||
|
self.mid_norm = LayerNorm(dim, **norm_kwargs) # This is a batch norm in the original but I don't like batch norm
|
||||||
|
self.swish = nn.SiLU()
|
||||||
|
self.pointwise_conv_2 = nn.Conv1d(dim, dim, kernel_size=1, bias=False)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = self.in_norm(x)
|
||||||
|
x = rearrange(x, 'b n d -> b d n')
|
||||||
|
x = self.pointwise_conv(x)
|
||||||
|
x = rearrange(x, 'b d n -> b n d')
|
||||||
|
x = self.glu(x)
|
||||||
|
x = rearrange(x, 'b n d -> b d n')
|
||||||
|
x = self.depthwise_conv(x)
|
||||||
|
x = rearrange(x, 'b d n -> b n d')
|
||||||
|
x = self.mid_norm(x)
|
||||||
|
x = self.swish(x)
|
||||||
|
x = rearrange(x, 'b n d -> b d n')
|
||||||
|
x = self.pointwise_conv_2(x)
|
||||||
|
x = rearrange(x, 'b d n -> b n d')
|
||||||
|
|
||||||
|
return x
|
||||||
|
|
||||||
|
class TransformerBlock(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim,
|
||||||
|
dim_heads = 64,
|
||||||
|
cross_attend = False,
|
||||||
|
dim_context = None,
|
||||||
|
global_cond_dim = None,
|
||||||
|
causal = False,
|
||||||
|
zero_init_branch_outputs = True,
|
||||||
|
conformer = False,
|
||||||
|
layer_ix = -1,
|
||||||
|
remove_norms = False,
|
||||||
|
attn_kwargs = {},
|
||||||
|
ff_kwargs = {},
|
||||||
|
norm_kwargs = {},
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
self.dim = dim
|
||||||
|
self.dim_heads = dim_heads
|
||||||
|
self.cross_attend = cross_attend
|
||||||
|
self.dim_context = dim_context
|
||||||
|
self.causal = causal
|
||||||
|
|
||||||
|
self.pre_norm = LayerNorm(dim, dtype=dtype, device=device, **norm_kwargs) if not remove_norms else nn.Identity()
|
||||||
|
|
||||||
|
self.self_attn = Attention(
|
||||||
|
dim,
|
||||||
|
dim_heads = dim_heads,
|
||||||
|
causal = causal,
|
||||||
|
zero_init_output=zero_init_branch_outputs,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations,
|
||||||
|
**attn_kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
if cross_attend:
|
||||||
|
self.cross_attend_norm = LayerNorm(dim, dtype=dtype, device=device, **norm_kwargs) if not remove_norms else nn.Identity()
|
||||||
|
self.cross_attn = Attention(
|
||||||
|
dim,
|
||||||
|
dim_heads = dim_heads,
|
||||||
|
dim_context=dim_context,
|
||||||
|
causal = causal,
|
||||||
|
zero_init_output=zero_init_branch_outputs,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations,
|
||||||
|
**attn_kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
self.ff_norm = LayerNorm(dim, dtype=dtype, device=device, **norm_kwargs) if not remove_norms else nn.Identity()
|
||||||
|
self.ff = FeedForward(dim, zero_init_output=zero_init_branch_outputs, dtype=dtype, device=device, operations=operations,**ff_kwargs)
|
||||||
|
|
||||||
|
self.layer_ix = layer_ix
|
||||||
|
|
||||||
|
self.conformer = ConformerModule(dim, norm_kwargs=norm_kwargs) if conformer else None
|
||||||
|
|
||||||
|
self.global_cond_dim = global_cond_dim
|
||||||
|
|
||||||
|
if global_cond_dim is not None:
|
||||||
|
self.to_scale_shift_gate = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
nn.Linear(global_cond_dim, dim * 6, bias=False)
|
||||||
|
)
|
||||||
|
|
||||||
|
nn.init.zeros_(self.to_scale_shift_gate[1].weight)
|
||||||
|
#nn.init.zeros_(self.to_scale_shift_gate_self[1].bias)
|
||||||
|
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
context = None,
|
||||||
|
global_cond=None,
|
||||||
|
mask = None,
|
||||||
|
context_mask = None,
|
||||||
|
rotary_pos_emb = None
|
||||||
|
):
|
||||||
|
if self.global_cond_dim is not None and self.global_cond_dim > 0 and global_cond is not None:
|
||||||
|
|
||||||
|
scale_self, shift_self, gate_self, scale_ff, shift_ff, gate_ff = self.to_scale_shift_gate(global_cond).unsqueeze(1).chunk(6, dim = -1)
|
||||||
|
|
||||||
|
# self-attention with adaLN
|
||||||
|
residual = x
|
||||||
|
x = self.pre_norm(x)
|
||||||
|
x = x * (1 + scale_self) + shift_self
|
||||||
|
x = self.self_attn(x, mask = mask, rotary_pos_emb = rotary_pos_emb)
|
||||||
|
x = x * torch.sigmoid(1 - gate_self)
|
||||||
|
x = x + residual
|
||||||
|
|
||||||
|
if context is not None:
|
||||||
|
x = x + self.cross_attn(self.cross_attend_norm(x), context = context, context_mask = context_mask)
|
||||||
|
|
||||||
|
if self.conformer is not None:
|
||||||
|
x = x + self.conformer(x)
|
||||||
|
|
||||||
|
# feedforward with adaLN
|
||||||
|
residual = x
|
||||||
|
x = self.ff_norm(x)
|
||||||
|
x = x * (1 + scale_ff) + shift_ff
|
||||||
|
x = self.ff(x)
|
||||||
|
x = x * torch.sigmoid(1 - gate_ff)
|
||||||
|
x = x + residual
|
||||||
|
|
||||||
|
else:
|
||||||
|
x = x + self.self_attn(self.pre_norm(x), mask = mask, rotary_pos_emb = rotary_pos_emb)
|
||||||
|
|
||||||
|
if context is not None:
|
||||||
|
x = x + self.cross_attn(self.cross_attend_norm(x), context = context, context_mask = context_mask)
|
||||||
|
|
||||||
|
if self.conformer is not None:
|
||||||
|
x = x + self.conformer(x)
|
||||||
|
|
||||||
|
x = x + self.ff(self.ff_norm(x))
|
||||||
|
|
||||||
|
return x
|
||||||
|
|
||||||
|
class ContinuousTransformer(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim,
|
||||||
|
depth,
|
||||||
|
*,
|
||||||
|
dim_in = None,
|
||||||
|
dim_out = None,
|
||||||
|
dim_heads = 64,
|
||||||
|
cross_attend=False,
|
||||||
|
cond_token_dim=None,
|
||||||
|
global_cond_dim=None,
|
||||||
|
causal=False,
|
||||||
|
rotary_pos_emb=True,
|
||||||
|
zero_init_branch_outputs=True,
|
||||||
|
conformer=False,
|
||||||
|
use_sinusoidal_emb=False,
|
||||||
|
use_abs_pos_emb=False,
|
||||||
|
abs_pos_emb_max_length=10000,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
**kwargs
|
||||||
|
):
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.dim = dim
|
||||||
|
self.depth = depth
|
||||||
|
self.causal = causal
|
||||||
|
self.layers = nn.ModuleList([])
|
||||||
|
|
||||||
|
self.project_in = operations.Linear(dim_in, dim, bias=False, dtype=dtype, device=device) if dim_in is not None else nn.Identity()
|
||||||
|
self.project_out = operations.Linear(dim, dim_out, bias=False, dtype=dtype, device=device) if dim_out is not None else nn.Identity()
|
||||||
|
|
||||||
|
if rotary_pos_emb:
|
||||||
|
self.rotary_pos_emb = RotaryEmbedding(max(dim_heads // 2, 32))
|
||||||
|
else:
|
||||||
|
self.rotary_pos_emb = None
|
||||||
|
|
||||||
|
self.use_sinusoidal_emb = use_sinusoidal_emb
|
||||||
|
if use_sinusoidal_emb:
|
||||||
|
self.pos_emb = ScaledSinusoidalEmbedding(dim)
|
||||||
|
|
||||||
|
self.use_abs_pos_emb = use_abs_pos_emb
|
||||||
|
if use_abs_pos_emb:
|
||||||
|
self.pos_emb = AbsolutePositionalEmbedding(dim, abs_pos_emb_max_length)
|
||||||
|
|
||||||
|
for i in range(depth):
|
||||||
|
self.layers.append(
|
||||||
|
TransformerBlock(
|
||||||
|
dim,
|
||||||
|
dim_heads = dim_heads,
|
||||||
|
cross_attend = cross_attend,
|
||||||
|
dim_context = cond_token_dim,
|
||||||
|
global_cond_dim = global_cond_dim,
|
||||||
|
causal = causal,
|
||||||
|
zero_init_branch_outputs = zero_init_branch_outputs,
|
||||||
|
conformer=conformer,
|
||||||
|
layer_ix=i,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
mask = None,
|
||||||
|
prepend_embeds = None,
|
||||||
|
prepend_mask = None,
|
||||||
|
global_cond = None,
|
||||||
|
return_info = False,
|
||||||
|
**kwargs
|
||||||
|
):
|
||||||
|
batch, seq, device = *x.shape[:2], x.device
|
||||||
|
|
||||||
|
info = {
|
||||||
|
"hidden_states": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
x = self.project_in(x)
|
||||||
|
|
||||||
|
if prepend_embeds is not None:
|
||||||
|
prepend_length, prepend_dim = prepend_embeds.shape[1:]
|
||||||
|
|
||||||
|
assert prepend_dim == x.shape[-1], 'prepend dimension must match sequence dimension'
|
||||||
|
|
||||||
|
x = torch.cat((prepend_embeds, x), dim = -2)
|
||||||
|
|
||||||
|
if prepend_mask is not None or mask is not None:
|
||||||
|
mask = mask if mask is not None else torch.ones((batch, seq), device = device, dtype = torch.bool)
|
||||||
|
prepend_mask = prepend_mask if prepend_mask is not None else torch.ones((batch, prepend_length), device = device, dtype = torch.bool)
|
||||||
|
|
||||||
|
mask = torch.cat((prepend_mask, mask), dim = -1)
|
||||||
|
|
||||||
|
# Attention layers
|
||||||
|
|
||||||
|
if self.rotary_pos_emb is not None:
|
||||||
|
rotary_pos_emb = self.rotary_pos_emb.forward_from_seq_len(x.shape[1], dtype=x.dtype, device=x.device)
|
||||||
|
else:
|
||||||
|
rotary_pos_emb = None
|
||||||
|
|
||||||
|
if self.use_sinusoidal_emb or self.use_abs_pos_emb:
|
||||||
|
x = x + self.pos_emb(x)
|
||||||
|
|
||||||
|
# Iterate over the transformer layers
|
||||||
|
for layer in self.layers:
|
||||||
|
x = layer(x, rotary_pos_emb = rotary_pos_emb, global_cond=global_cond, **kwargs)
|
||||||
|
# x = checkpoint(layer, x, rotary_pos_emb = rotary_pos_emb, global_cond=global_cond, **kwargs)
|
||||||
|
|
||||||
|
if return_info:
|
||||||
|
info["hidden_states"].append(x)
|
||||||
|
|
||||||
|
x = self.project_out(x)
|
||||||
|
|
||||||
|
if return_info:
|
||||||
|
return x, info
|
||||||
|
|
||||||
|
return x
|
||||||
|
|
||||||
|
class AudioDiffusionTransformer(nn.Module):
|
||||||
|
def __init__(self,
|
||||||
|
io_channels=64,
|
||||||
|
patch_size=1,
|
||||||
|
embed_dim=1536,
|
||||||
|
cond_token_dim=768,
|
||||||
|
project_cond_tokens=False,
|
||||||
|
global_cond_dim=1536,
|
||||||
|
project_global_cond=True,
|
||||||
|
input_concat_dim=0,
|
||||||
|
prepend_cond_dim=0,
|
||||||
|
depth=24,
|
||||||
|
num_heads=24,
|
||||||
|
transformer_type: tp.Literal["continuous_transformer"] = "continuous_transformer",
|
||||||
|
global_cond_type: tp.Literal["prepend", "adaLN"] = "prepend",
|
||||||
|
audio_model="",
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
**kwargs):
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.dtype = dtype
|
||||||
|
self.cond_token_dim = cond_token_dim
|
||||||
|
|
||||||
|
# Timestep embeddings
|
||||||
|
timestep_features_dim = 256
|
||||||
|
|
||||||
|
self.timestep_features = FourierFeatures(1, timestep_features_dim, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.to_timestep_embed = nn.Sequential(
|
||||||
|
operations.Linear(timestep_features_dim, embed_dim, bias=True, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(embed_dim, embed_dim, bias=True, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
if cond_token_dim > 0:
|
||||||
|
# Conditioning tokens
|
||||||
|
|
||||||
|
cond_embed_dim = cond_token_dim if not project_cond_tokens else embed_dim
|
||||||
|
self.to_cond_embed = nn.Sequential(
|
||||||
|
operations.Linear(cond_token_dim, cond_embed_dim, bias=False, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(cond_embed_dim, cond_embed_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cond_embed_dim = 0
|
||||||
|
|
||||||
|
if global_cond_dim > 0:
|
||||||
|
# Global conditioning
|
||||||
|
global_embed_dim = global_cond_dim if not project_global_cond else embed_dim
|
||||||
|
self.to_global_embed = nn.Sequential(
|
||||||
|
operations.Linear(global_cond_dim, global_embed_dim, bias=False, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(global_embed_dim, global_embed_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
if prepend_cond_dim > 0:
|
||||||
|
# Prepend conditioning
|
||||||
|
self.to_prepend_embed = nn.Sequential(
|
||||||
|
operations.Linear(prepend_cond_dim, embed_dim, bias=False, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(embed_dim, embed_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
self.input_concat_dim = input_concat_dim
|
||||||
|
|
||||||
|
dim_in = io_channels + self.input_concat_dim
|
||||||
|
|
||||||
|
self.patch_size = patch_size
|
||||||
|
|
||||||
|
# Transformer
|
||||||
|
|
||||||
|
self.transformer_type = transformer_type
|
||||||
|
|
||||||
|
self.global_cond_type = global_cond_type
|
||||||
|
|
||||||
|
if self.transformer_type == "continuous_transformer":
|
||||||
|
|
||||||
|
global_dim = None
|
||||||
|
|
||||||
|
if self.global_cond_type == "adaLN":
|
||||||
|
# The global conditioning is projected to the embed_dim already at this point
|
||||||
|
global_dim = embed_dim
|
||||||
|
|
||||||
|
self.transformer = ContinuousTransformer(
|
||||||
|
dim=embed_dim,
|
||||||
|
depth=depth,
|
||||||
|
dim_heads=embed_dim // num_heads,
|
||||||
|
dim_in=dim_in * patch_size,
|
||||||
|
dim_out=io_channels * patch_size,
|
||||||
|
cross_attend = cond_token_dim > 0,
|
||||||
|
cond_token_dim = cond_embed_dim,
|
||||||
|
global_cond_dim=global_dim,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown transformer type: {self.transformer_type}")
|
||||||
|
|
||||||
|
self.preprocess_conv = operations.Conv1d(dim_in, dim_in, 1, bias=False, dtype=dtype, device=device)
|
||||||
|
self.postprocess_conv = operations.Conv1d(io_channels, io_channels, 1, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def _forward(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
t,
|
||||||
|
mask=None,
|
||||||
|
cross_attn_cond=None,
|
||||||
|
cross_attn_cond_mask=None,
|
||||||
|
input_concat_cond=None,
|
||||||
|
global_embed=None,
|
||||||
|
prepend_cond=None,
|
||||||
|
prepend_cond_mask=None,
|
||||||
|
return_info=False,
|
||||||
|
**kwargs):
|
||||||
|
|
||||||
|
if cross_attn_cond is not None:
|
||||||
|
cross_attn_cond = self.to_cond_embed(cross_attn_cond)
|
||||||
|
|
||||||
|
if global_embed is not None:
|
||||||
|
# Project the global conditioning to the embedding dimension
|
||||||
|
global_embed = self.to_global_embed(global_embed)
|
||||||
|
|
||||||
|
prepend_inputs = None
|
||||||
|
prepend_mask = None
|
||||||
|
prepend_length = 0
|
||||||
|
if prepend_cond is not None:
|
||||||
|
# Project the prepend conditioning to the embedding dimension
|
||||||
|
prepend_cond = self.to_prepend_embed(prepend_cond)
|
||||||
|
|
||||||
|
prepend_inputs = prepend_cond
|
||||||
|
if prepend_cond_mask is not None:
|
||||||
|
prepend_mask = prepend_cond_mask
|
||||||
|
|
||||||
|
if input_concat_cond is not None:
|
||||||
|
|
||||||
|
# Interpolate input_concat_cond to the same length as x
|
||||||
|
if input_concat_cond.shape[2] != x.shape[2]:
|
||||||
|
input_concat_cond = F.interpolate(input_concat_cond, (x.shape[2], ), mode='nearest')
|
||||||
|
|
||||||
|
x = torch.cat([x, input_concat_cond], dim=1)
|
||||||
|
|
||||||
|
# Get the batch of timestep embeddings
|
||||||
|
timestep_embed = self.to_timestep_embed(self.timestep_features(t[:, None]).to(x.dtype)) # (b, embed_dim)
|
||||||
|
|
||||||
|
# Timestep embedding is considered a global embedding. Add to the global conditioning if it exists
|
||||||
|
if global_embed is not None:
|
||||||
|
global_embed = global_embed + timestep_embed
|
||||||
|
else:
|
||||||
|
global_embed = timestep_embed
|
||||||
|
|
||||||
|
# Add the global_embed to the prepend inputs if there is no global conditioning support in the transformer
|
||||||
|
if self.global_cond_type == "prepend":
|
||||||
|
if prepend_inputs is None:
|
||||||
|
# Prepend inputs are just the global embed, and the mask is all ones
|
||||||
|
prepend_inputs = global_embed.unsqueeze(1)
|
||||||
|
prepend_mask = torch.ones((x.shape[0], 1), device=x.device, dtype=torch.bool)
|
||||||
|
else:
|
||||||
|
# Prepend inputs are the prepend conditioning + the global embed
|
||||||
|
prepend_inputs = torch.cat([prepend_inputs, global_embed.unsqueeze(1)], dim=1)
|
||||||
|
prepend_mask = torch.cat([prepend_mask, torch.ones((x.shape[0], 1), device=x.device, dtype=torch.bool)], dim=1)
|
||||||
|
|
||||||
|
prepend_length = prepend_inputs.shape[1]
|
||||||
|
|
||||||
|
x = self.preprocess_conv(x) + x
|
||||||
|
|
||||||
|
x = rearrange(x, "b c t -> b t c")
|
||||||
|
|
||||||
|
extra_args = {}
|
||||||
|
|
||||||
|
if self.global_cond_type == "adaLN":
|
||||||
|
extra_args["global_cond"] = global_embed
|
||||||
|
|
||||||
|
if self.patch_size > 1:
|
||||||
|
x = rearrange(x, "b (t p) c -> b t (c p)", p=self.patch_size)
|
||||||
|
|
||||||
|
if self.transformer_type == "x-transformers":
|
||||||
|
output = self.transformer(x, prepend_embeds=prepend_inputs, context=cross_attn_cond, context_mask=cross_attn_cond_mask, mask=mask, prepend_mask=prepend_mask, **extra_args, **kwargs)
|
||||||
|
elif self.transformer_type == "continuous_transformer":
|
||||||
|
output = self.transformer(x, prepend_embeds=prepend_inputs, context=cross_attn_cond, context_mask=cross_attn_cond_mask, mask=mask, prepend_mask=prepend_mask, return_info=return_info, **extra_args, **kwargs)
|
||||||
|
|
||||||
|
if return_info:
|
||||||
|
output, info = output
|
||||||
|
elif self.transformer_type == "mm_transformer":
|
||||||
|
output = self.transformer(x, context=cross_attn_cond, mask=mask, context_mask=cross_attn_cond_mask, **extra_args, **kwargs)
|
||||||
|
|
||||||
|
output = rearrange(output, "b t c -> b c t")[:,:,prepend_length:]
|
||||||
|
|
||||||
|
if self.patch_size > 1:
|
||||||
|
output = rearrange(output, "b (c p) t -> b c (t p)", p=self.patch_size)
|
||||||
|
|
||||||
|
output = self.postprocess_conv(output) + output
|
||||||
|
|
||||||
|
if return_info:
|
||||||
|
return output, info
|
||||||
|
|
||||||
|
return output
|
||||||
|
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
timestep,
|
||||||
|
context=None,
|
||||||
|
context_mask=None,
|
||||||
|
input_concat_cond=None,
|
||||||
|
global_embed=None,
|
||||||
|
negative_global_embed=None,
|
||||||
|
prepend_cond=None,
|
||||||
|
prepend_cond_mask=None,
|
||||||
|
mask=None,
|
||||||
|
return_info=False,
|
||||||
|
control=None,
|
||||||
|
transformer_options={},
|
||||||
|
**kwargs):
|
||||||
|
return self._forward(
|
||||||
|
x,
|
||||||
|
timestep,
|
||||||
|
cross_attn_cond=context,
|
||||||
|
cross_attn_cond_mask=context_mask,
|
||||||
|
input_concat_cond=input_concat_cond,
|
||||||
|
global_embed=global_embed,
|
||||||
|
prepend_cond=prepend_cond,
|
||||||
|
prepend_cond_mask=prepend_cond_mask,
|
||||||
|
mask=mask,
|
||||||
|
return_info=return_info,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
108
comfy/ldm/audio/embedders.py
Normal file
108
comfy/ldm/audio/embedders.py
Normal file
@ -0,0 +1,108 @@
|
|||||||
|
# code adapted from: https://github.com/Stability-AI/stable-audio-tools
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
from torch import Tensor, einsum
|
||||||
|
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, TypeVar, Union
|
||||||
|
from einops import rearrange
|
||||||
|
import math
|
||||||
|
import comfy.ops
|
||||||
|
|
||||||
|
class LearnedPositionalEmbedding(nn.Module):
|
||||||
|
"""Used for continuous time"""
|
||||||
|
|
||||||
|
def __init__(self, dim: int):
|
||||||
|
super().__init__()
|
||||||
|
assert (dim % 2) == 0
|
||||||
|
half_dim = dim // 2
|
||||||
|
self.weights = nn.Parameter(torch.empty(half_dim))
|
||||||
|
|
||||||
|
def forward(self, x: Tensor) -> Tensor:
|
||||||
|
x = rearrange(x, "b -> b 1")
|
||||||
|
freqs = x * rearrange(self.weights, "d -> 1 d") * 2 * math.pi
|
||||||
|
fouriered = torch.cat((freqs.sin(), freqs.cos()), dim=-1)
|
||||||
|
fouriered = torch.cat((x, fouriered), dim=-1)
|
||||||
|
return fouriered
|
||||||
|
|
||||||
|
def TimePositionalEmbedding(dim: int, out_features: int) -> nn.Module:
|
||||||
|
return nn.Sequential(
|
||||||
|
LearnedPositionalEmbedding(dim),
|
||||||
|
comfy.ops.manual_cast.Linear(in_features=dim + 1, out_features=out_features),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class NumberEmbedder(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
features: int,
|
||||||
|
dim: int = 256,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.features = features
|
||||||
|
self.embedding = TimePositionalEmbedding(dim=dim, out_features=features)
|
||||||
|
|
||||||
|
def forward(self, x: Union[List[float], Tensor]) -> Tensor:
|
||||||
|
if not torch.is_tensor(x):
|
||||||
|
device = next(self.embedding.parameters()).device
|
||||||
|
x = torch.tensor(x, device=device)
|
||||||
|
assert isinstance(x, Tensor)
|
||||||
|
shape = x.shape
|
||||||
|
x = rearrange(x, "... -> (...)")
|
||||||
|
embedding = self.embedding(x)
|
||||||
|
x = embedding.view(*shape, self.features)
|
||||||
|
return x # type: ignore
|
||||||
|
|
||||||
|
|
||||||
|
class Conditioner(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim: int,
|
||||||
|
output_dim: int,
|
||||||
|
project_out: bool = False
|
||||||
|
):
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.dim = dim
|
||||||
|
self.output_dim = output_dim
|
||||||
|
self.proj_out = nn.Linear(dim, output_dim) if (dim != output_dim or project_out) else nn.Identity()
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
raise NotImplementedError()
|
||||||
|
|
||||||
|
class NumberConditioner(Conditioner):
|
||||||
|
'''
|
||||||
|
Conditioner that takes a list of floats, normalizes them for a given range, and returns a list of embeddings
|
||||||
|
'''
|
||||||
|
def __init__(self,
|
||||||
|
output_dim: int,
|
||||||
|
min_val: float=0,
|
||||||
|
max_val: float=1
|
||||||
|
):
|
||||||
|
super().__init__(output_dim, output_dim)
|
||||||
|
|
||||||
|
self.min_val = min_val
|
||||||
|
self.max_val = max_val
|
||||||
|
|
||||||
|
self.embedder = NumberEmbedder(features=output_dim)
|
||||||
|
|
||||||
|
def forward(self, floats, device=None):
|
||||||
|
# Cast the inputs to floats
|
||||||
|
floats = [float(x) for x in floats]
|
||||||
|
|
||||||
|
if device is None:
|
||||||
|
device = next(self.embedder.parameters()).device
|
||||||
|
|
||||||
|
floats = torch.tensor(floats).to(device)
|
||||||
|
|
||||||
|
floats = floats.clamp(self.min_val, self.max_val)
|
||||||
|
|
||||||
|
normalized_floats = (floats - self.min_val) / (self.max_val - self.min_val)
|
||||||
|
|
||||||
|
# Cast floats to same type as embedder
|
||||||
|
embedder_dtype = next(self.embedder.parameters()).dtype
|
||||||
|
normalized_floats = normalized_floats.to(embedder_dtype)
|
||||||
|
|
||||||
|
float_embeds = self.embedder(normalized_floats).unsqueeze(1)
|
||||||
|
|
||||||
|
return [float_embeds, torch.ones(float_embeds.shape[0], 1).to(device)]
|
||||||
479
comfy/ldm/aura/mmdit.py
Normal file
479
comfy/ldm/aura/mmdit.py
Normal file
@ -0,0 +1,479 @@
|
|||||||
|
#AuraFlow MMDiT
|
||||||
|
#Originally written by the AuraFlow Authors
|
||||||
|
|
||||||
|
import math
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
import torch.nn.functional as F
|
||||||
|
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention
|
||||||
|
|
||||||
|
def modulate(x, shift, scale):
|
||||||
|
return x * (1 + scale.unsqueeze(1)) + shift.unsqueeze(1)
|
||||||
|
|
||||||
|
|
||||||
|
def find_multiple(n: int, k: int) -> int:
|
||||||
|
if n % k == 0:
|
||||||
|
return n
|
||||||
|
return n + k - (n % k)
|
||||||
|
|
||||||
|
|
||||||
|
class MLP(nn.Module):
|
||||||
|
def __init__(self, dim, hidden_dim=None, dtype=None, device=None, operations=None) -> None:
|
||||||
|
super().__init__()
|
||||||
|
if hidden_dim is None:
|
||||||
|
hidden_dim = 4 * dim
|
||||||
|
|
||||||
|
n_hidden = int(2 * hidden_dim / 3)
|
||||||
|
n_hidden = find_multiple(n_hidden, 256)
|
||||||
|
|
||||||
|
self.c_fc1 = operations.Linear(dim, n_hidden, bias=False, dtype=dtype, device=device)
|
||||||
|
self.c_fc2 = operations.Linear(dim, n_hidden, bias=False, dtype=dtype, device=device)
|
||||||
|
self.c_proj = operations.Linear(n_hidden, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
x = F.silu(self.c_fc1(x)) * self.c_fc2(x)
|
||||||
|
x = self.c_proj(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class MultiHeadLayerNorm(nn.Module):
|
||||||
|
def __init__(self, hidden_size=None, eps=1e-5, dtype=None, device=None):
|
||||||
|
# Copy pasta from https://github.com/huggingface/transformers/blob/e5f71ecaae50ea476d1e12351003790273c4b2ed/src/transformers/models/cohere/modeling_cohere.py#L78
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
self.weight = nn.Parameter(torch.empty(hidden_size, dtype=dtype, device=device))
|
||||||
|
self.variance_epsilon = eps
|
||||||
|
|
||||||
|
def forward(self, hidden_states):
|
||||||
|
input_dtype = hidden_states.dtype
|
||||||
|
hidden_states = hidden_states.to(torch.float32)
|
||||||
|
mean = hidden_states.mean(-1, keepdim=True)
|
||||||
|
variance = (hidden_states - mean).pow(2).mean(-1, keepdim=True)
|
||||||
|
hidden_states = (hidden_states - mean) * torch.rsqrt(
|
||||||
|
variance + self.variance_epsilon
|
||||||
|
)
|
||||||
|
hidden_states = self.weight.to(torch.float32) * hidden_states
|
||||||
|
return hidden_states.to(input_dtype)
|
||||||
|
|
||||||
|
class SingleAttention(nn.Module):
|
||||||
|
def __init__(self, dim, n_heads, mh_qknorm=False, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.n_heads = n_heads
|
||||||
|
self.head_dim = dim // n_heads
|
||||||
|
|
||||||
|
# this is for cond
|
||||||
|
self.w1q = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w1k = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w1v = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w1o = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.q_norm1 = (
|
||||||
|
MultiHeadLayerNorm((self.n_heads, self.head_dim), dtype=dtype, device=device)
|
||||||
|
if mh_qknorm
|
||||||
|
else operations.LayerNorm(self.head_dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
self.k_norm1 = (
|
||||||
|
MultiHeadLayerNorm((self.n_heads, self.head_dim), dtype=dtype, device=device)
|
||||||
|
if mh_qknorm
|
||||||
|
else operations.LayerNorm(self.head_dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
#@torch.compile()
|
||||||
|
def forward(self, c):
|
||||||
|
|
||||||
|
bsz, seqlen1, _ = c.shape
|
||||||
|
|
||||||
|
q, k, v = self.w1q(c), self.w1k(c), self.w1v(c)
|
||||||
|
q = q.view(bsz, seqlen1, self.n_heads, self.head_dim)
|
||||||
|
k = k.view(bsz, seqlen1, self.n_heads, self.head_dim)
|
||||||
|
v = v.view(bsz, seqlen1, self.n_heads, self.head_dim)
|
||||||
|
q, k = self.q_norm1(q), self.k_norm1(k)
|
||||||
|
|
||||||
|
output = optimized_attention(q.permute(0, 2, 1, 3), k.permute(0, 2, 1, 3), v.permute(0, 2, 1, 3), self.n_heads, skip_reshape=True)
|
||||||
|
c = self.w1o(output)
|
||||||
|
return c
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class DoubleAttention(nn.Module):
|
||||||
|
def __init__(self, dim, n_heads, mh_qknorm=False, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.n_heads = n_heads
|
||||||
|
self.head_dim = dim // n_heads
|
||||||
|
|
||||||
|
# this is for cond
|
||||||
|
self.w1q = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w1k = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w1v = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w1o = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
# this is for x
|
||||||
|
self.w2q = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w2k = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w2v = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.w2o = operations.Linear(dim, dim, bias=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.q_norm1 = (
|
||||||
|
MultiHeadLayerNorm((self.n_heads, self.head_dim), dtype=dtype, device=device)
|
||||||
|
if mh_qknorm
|
||||||
|
else operations.LayerNorm(self.head_dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
self.k_norm1 = (
|
||||||
|
MultiHeadLayerNorm((self.n_heads, self.head_dim), dtype=dtype, device=device)
|
||||||
|
if mh_qknorm
|
||||||
|
else operations.LayerNorm(self.head_dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
self.q_norm2 = (
|
||||||
|
MultiHeadLayerNorm((self.n_heads, self.head_dim), dtype=dtype, device=device)
|
||||||
|
if mh_qknorm
|
||||||
|
else operations.LayerNorm(self.head_dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
self.k_norm2 = (
|
||||||
|
MultiHeadLayerNorm((self.n_heads, self.head_dim), dtype=dtype, device=device)
|
||||||
|
if mh_qknorm
|
||||||
|
else operations.LayerNorm(self.head_dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
#@torch.compile()
|
||||||
|
def forward(self, c, x):
|
||||||
|
|
||||||
|
bsz, seqlen1, _ = c.shape
|
||||||
|
bsz, seqlen2, _ = x.shape
|
||||||
|
seqlen = seqlen1 + seqlen2
|
||||||
|
|
||||||
|
cq, ck, cv = self.w1q(c), self.w1k(c), self.w1v(c)
|
||||||
|
cq = cq.view(bsz, seqlen1, self.n_heads, self.head_dim)
|
||||||
|
ck = ck.view(bsz, seqlen1, self.n_heads, self.head_dim)
|
||||||
|
cv = cv.view(bsz, seqlen1, self.n_heads, self.head_dim)
|
||||||
|
cq, ck = self.q_norm1(cq), self.k_norm1(ck)
|
||||||
|
|
||||||
|
xq, xk, xv = self.w2q(x), self.w2k(x), self.w2v(x)
|
||||||
|
xq = xq.view(bsz, seqlen2, self.n_heads, self.head_dim)
|
||||||
|
xk = xk.view(bsz, seqlen2, self.n_heads, self.head_dim)
|
||||||
|
xv = xv.view(bsz, seqlen2, self.n_heads, self.head_dim)
|
||||||
|
xq, xk = self.q_norm2(xq), self.k_norm2(xk)
|
||||||
|
|
||||||
|
# concat all
|
||||||
|
q, k, v = (
|
||||||
|
torch.cat([cq, xq], dim=1),
|
||||||
|
torch.cat([ck, xk], dim=1),
|
||||||
|
torch.cat([cv, xv], dim=1),
|
||||||
|
)
|
||||||
|
|
||||||
|
output = optimized_attention(q.permute(0, 2, 1, 3), k.permute(0, 2, 1, 3), v.permute(0, 2, 1, 3), self.n_heads, skip_reshape=True)
|
||||||
|
|
||||||
|
c, x = output.split([seqlen1, seqlen2], dim=1)
|
||||||
|
c = self.w1o(c)
|
||||||
|
x = self.w2o(x)
|
||||||
|
|
||||||
|
return c, x
|
||||||
|
|
||||||
|
|
||||||
|
class MMDiTBlock(nn.Module):
|
||||||
|
def __init__(self, dim, heads=8, global_conddim=1024, is_last=False, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.normC1 = operations.LayerNorm(dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
self.normC2 = operations.LayerNorm(dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
if not is_last:
|
||||||
|
self.mlpC = MLP(dim, hidden_dim=dim * 4, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.modC = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(global_conddim, 6 * dim, bias=False, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.modC = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(global_conddim, 2 * dim, bias=False, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.normX1 = operations.LayerNorm(dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
self.normX2 = operations.LayerNorm(dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
self.mlpX = MLP(dim, hidden_dim=dim * 4, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.modX = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(global_conddim, 6 * dim, bias=False, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.attn = DoubleAttention(dim, heads, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.is_last = is_last
|
||||||
|
|
||||||
|
#@torch.compile()
|
||||||
|
def forward(self, c, x, global_cond, **kwargs):
|
||||||
|
|
||||||
|
cres, xres = c, x
|
||||||
|
|
||||||
|
cshift_msa, cscale_msa, cgate_msa, cshift_mlp, cscale_mlp, cgate_mlp = (
|
||||||
|
self.modC(global_cond).chunk(6, dim=1)
|
||||||
|
)
|
||||||
|
|
||||||
|
c = modulate(self.normC1(c), cshift_msa, cscale_msa)
|
||||||
|
|
||||||
|
# xpath
|
||||||
|
xshift_msa, xscale_msa, xgate_msa, xshift_mlp, xscale_mlp, xgate_mlp = (
|
||||||
|
self.modX(global_cond).chunk(6, dim=1)
|
||||||
|
)
|
||||||
|
|
||||||
|
x = modulate(self.normX1(x), xshift_msa, xscale_msa)
|
||||||
|
|
||||||
|
# attention
|
||||||
|
c, x = self.attn(c, x)
|
||||||
|
|
||||||
|
|
||||||
|
c = self.normC2(cres + cgate_msa.unsqueeze(1) * c)
|
||||||
|
c = cgate_mlp.unsqueeze(1) * self.mlpC(modulate(c, cshift_mlp, cscale_mlp))
|
||||||
|
c = cres + c
|
||||||
|
|
||||||
|
x = self.normX2(xres + xgate_msa.unsqueeze(1) * x)
|
||||||
|
x = xgate_mlp.unsqueeze(1) * self.mlpX(modulate(x, xshift_mlp, xscale_mlp))
|
||||||
|
x = xres + x
|
||||||
|
|
||||||
|
return c, x
|
||||||
|
|
||||||
|
class DiTBlock(nn.Module):
|
||||||
|
# like MMDiTBlock, but it only has X
|
||||||
|
def __init__(self, dim, heads=8, global_conddim=1024, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.norm1 = operations.LayerNorm(dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
self.norm2 = operations.LayerNorm(dim, elementwise_affine=False, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.modCX = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(global_conddim, 6 * dim, bias=False, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.attn = SingleAttention(dim, heads, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.mlp = MLP(dim, hidden_dim=dim * 4, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
#@torch.compile()
|
||||||
|
def forward(self, cx, global_cond, **kwargs):
|
||||||
|
cxres = cx
|
||||||
|
shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.modCX(
|
||||||
|
global_cond
|
||||||
|
).chunk(6, dim=1)
|
||||||
|
cx = modulate(self.norm1(cx), shift_msa, scale_msa)
|
||||||
|
cx = self.attn(cx)
|
||||||
|
cx = self.norm2(cxres + gate_msa.unsqueeze(1) * cx)
|
||||||
|
mlpout = self.mlp(modulate(cx, shift_mlp, scale_mlp))
|
||||||
|
cx = gate_mlp.unsqueeze(1) * mlpout
|
||||||
|
|
||||||
|
cx = cxres + cx
|
||||||
|
|
||||||
|
return cx
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class TimestepEmbedder(nn.Module):
|
||||||
|
def __init__(self, hidden_size, frequency_embedding_size=256, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.mlp = nn.Sequential(
|
||||||
|
operations.Linear(frequency_embedding_size, hidden_size, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(hidden_size, hidden_size, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
self.frequency_embedding_size = frequency_embedding_size
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def timestep_embedding(t, dim, max_period=10000):
|
||||||
|
half = dim // 2
|
||||||
|
freqs = 1000 * torch.exp(
|
||||||
|
-math.log(max_period) * torch.arange(start=0, end=half) / half
|
||||||
|
).to(t.device)
|
||||||
|
args = t[:, None] * freqs[None]
|
||||||
|
embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)
|
||||||
|
if dim % 2:
|
||||||
|
embedding = torch.cat(
|
||||||
|
[embedding, torch.zeros_like(embedding[:, :1])], dim=-1
|
||||||
|
)
|
||||||
|
return embedding
|
||||||
|
|
||||||
|
#@torch.compile()
|
||||||
|
def forward(self, t, dtype):
|
||||||
|
t_freq = self.timestep_embedding(t, self.frequency_embedding_size).to(dtype)
|
||||||
|
t_emb = self.mlp(t_freq)
|
||||||
|
return t_emb
|
||||||
|
|
||||||
|
|
||||||
|
class MMDiT(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
in_channels=4,
|
||||||
|
out_channels=4,
|
||||||
|
patch_size=2,
|
||||||
|
dim=3072,
|
||||||
|
n_layers=36,
|
||||||
|
n_double_layers=4,
|
||||||
|
n_heads=12,
|
||||||
|
global_conddim=3072,
|
||||||
|
cond_seq_dim=2048,
|
||||||
|
max_seq=32 * 32,
|
||||||
|
device=None,
|
||||||
|
dtype=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.dtype = dtype
|
||||||
|
|
||||||
|
self.t_embedder = TimestepEmbedder(global_conddim, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
self.cond_seq_linear = operations.Linear(
|
||||||
|
cond_seq_dim, dim, bias=False, dtype=dtype, device=device
|
||||||
|
) # linear for something like text sequence.
|
||||||
|
self.init_x_linear = operations.Linear(
|
||||||
|
patch_size * patch_size * in_channels, dim, dtype=dtype, device=device
|
||||||
|
) # init linear for patchified image.
|
||||||
|
|
||||||
|
self.positional_encoding = nn.Parameter(torch.empty(1, max_seq, dim, dtype=dtype, device=device))
|
||||||
|
self.register_tokens = nn.Parameter(torch.empty(1, 8, dim, dtype=dtype, device=device))
|
||||||
|
|
||||||
|
self.double_layers = nn.ModuleList([])
|
||||||
|
self.single_layers = nn.ModuleList([])
|
||||||
|
|
||||||
|
|
||||||
|
for idx in range(n_double_layers):
|
||||||
|
self.double_layers.append(
|
||||||
|
MMDiTBlock(dim, n_heads, global_conddim, is_last=(idx == n_layers - 1), dtype=dtype, device=device, operations=operations)
|
||||||
|
)
|
||||||
|
|
||||||
|
for idx in range(n_double_layers, n_layers):
|
||||||
|
self.single_layers.append(
|
||||||
|
DiTBlock(dim, n_heads, global_conddim, dtype=dtype, device=device, operations=operations)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
self.final_linear = operations.Linear(
|
||||||
|
dim, patch_size * patch_size * out_channels, bias=False, dtype=dtype, device=device
|
||||||
|
)
|
||||||
|
|
||||||
|
self.modF = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(global_conddim, 2 * dim, bias=False, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.out_channels = out_channels
|
||||||
|
self.patch_size = patch_size
|
||||||
|
self.n_double_layers = n_double_layers
|
||||||
|
self.n_layers = n_layers
|
||||||
|
|
||||||
|
self.h_max = round(max_seq**0.5)
|
||||||
|
self.w_max = round(max_seq**0.5)
|
||||||
|
|
||||||
|
@torch.no_grad()
|
||||||
|
def extend_pe(self, init_dim=(16, 16), target_dim=(64, 64)):
|
||||||
|
# extend pe
|
||||||
|
pe_data = self.positional_encoding.data.squeeze(0)[: init_dim[0] * init_dim[1]]
|
||||||
|
|
||||||
|
pe_as_2d = pe_data.view(init_dim[0], init_dim[1], -1).permute(2, 0, 1)
|
||||||
|
|
||||||
|
# now we need to extend this to target_dim. for this we will use interpolation.
|
||||||
|
# we will use torch.nn.functional.interpolate
|
||||||
|
pe_as_2d = F.interpolate(
|
||||||
|
pe_as_2d.unsqueeze(0), size=target_dim, mode="bilinear"
|
||||||
|
)
|
||||||
|
pe_new = pe_as_2d.squeeze(0).permute(1, 2, 0).flatten(0, 1)
|
||||||
|
self.positional_encoding.data = pe_new.unsqueeze(0).contiguous()
|
||||||
|
self.h_max, self.w_max = target_dim
|
||||||
|
print("PE extended to", target_dim)
|
||||||
|
|
||||||
|
def pe_selection_index_based_on_dim(self, h, w):
|
||||||
|
h_p, w_p = h // self.patch_size, w // self.patch_size
|
||||||
|
original_pe_indexes = torch.arange(self.positional_encoding.shape[1])
|
||||||
|
original_pe_indexes = original_pe_indexes.view(self.h_max, self.w_max)
|
||||||
|
starth = self.h_max // 2 - h_p // 2
|
||||||
|
endh =starth + h_p
|
||||||
|
startw = self.w_max // 2 - w_p // 2
|
||||||
|
endw = startw + w_p
|
||||||
|
original_pe_indexes = original_pe_indexes[
|
||||||
|
starth:endh, startw:endw
|
||||||
|
]
|
||||||
|
return original_pe_indexes.flatten()
|
||||||
|
|
||||||
|
def unpatchify(self, x, h, w):
|
||||||
|
c = self.out_channels
|
||||||
|
p = self.patch_size
|
||||||
|
|
||||||
|
x = x.reshape(shape=(x.shape[0], h, w, p, p, c))
|
||||||
|
x = torch.einsum("nhwpqc->nchpwq", x)
|
||||||
|
imgs = x.reshape(shape=(x.shape[0], c, h * p, w * p))
|
||||||
|
return imgs
|
||||||
|
|
||||||
|
def patchify(self, x):
|
||||||
|
B, C, H, W = x.size()
|
||||||
|
pad_h = (self.patch_size - H % self.patch_size) % self.patch_size
|
||||||
|
pad_w = (self.patch_size - W % self.patch_size) % self.patch_size
|
||||||
|
|
||||||
|
x = torch.nn.functional.pad(x, (0, pad_w, 0, pad_h), mode='reflect')
|
||||||
|
x = x.view(
|
||||||
|
B,
|
||||||
|
C,
|
||||||
|
(H + 1) // self.patch_size,
|
||||||
|
self.patch_size,
|
||||||
|
(W + 1) // self.patch_size,
|
||||||
|
self.patch_size,
|
||||||
|
)
|
||||||
|
x = x.permute(0, 2, 4, 1, 3, 5).flatten(-3).flatten(1, 2)
|
||||||
|
return x
|
||||||
|
|
||||||
|
def apply_pos_embeds(self, x, h, w):
|
||||||
|
h = (h + 1) // self.patch_size
|
||||||
|
w = (w + 1) // self.patch_size
|
||||||
|
max_dim = max(h, w)
|
||||||
|
|
||||||
|
cur_dim = self.h_max
|
||||||
|
pos_encoding = self.positional_encoding.reshape(1, cur_dim, cur_dim, -1).to(device=x.device, dtype=x.dtype)
|
||||||
|
|
||||||
|
if max_dim > cur_dim:
|
||||||
|
pos_encoding = F.interpolate(pos_encoding.movedim(-1, 1), (max_dim, max_dim), mode="bilinear").movedim(1, -1)
|
||||||
|
cur_dim = max_dim
|
||||||
|
|
||||||
|
from_h = (cur_dim - h) // 2
|
||||||
|
from_w = (cur_dim - w) // 2
|
||||||
|
pos_encoding = pos_encoding[:,from_h:from_h+h,from_w:from_w+w]
|
||||||
|
return x + pos_encoding.reshape(1, -1, self.positional_encoding.shape[-1])
|
||||||
|
|
||||||
|
def forward(self, x, timestep, context, **kwargs):
|
||||||
|
# patchify x, add PE
|
||||||
|
b, c, h, w = x.shape
|
||||||
|
|
||||||
|
# pe_indexes = self.pe_selection_index_based_on_dim(h, w)
|
||||||
|
# print(pe_indexes, pe_indexes.shape)
|
||||||
|
|
||||||
|
x = self.init_x_linear(self.patchify(x)) # B, T_x, D
|
||||||
|
x = self.apply_pos_embeds(x, h, w)
|
||||||
|
# x = x + self.positional_encoding[:, : x.size(1)].to(device=x.device, dtype=x.dtype)
|
||||||
|
# x = x + self.positional_encoding[:, pe_indexes].to(device=x.device, dtype=x.dtype)
|
||||||
|
|
||||||
|
# process conditions for MMDiT Blocks
|
||||||
|
c_seq = context # B, T_c, D_c
|
||||||
|
t = timestep
|
||||||
|
|
||||||
|
c = self.cond_seq_linear(c_seq) # B, T_c, D
|
||||||
|
c = torch.cat([self.register_tokens.to(device=c.device, dtype=c.dtype).repeat(c.size(0), 1, 1), c], dim=1)
|
||||||
|
|
||||||
|
global_cond = self.t_embedder(t, x.dtype) # B, D
|
||||||
|
|
||||||
|
if len(self.double_layers) > 0:
|
||||||
|
for layer in self.double_layers:
|
||||||
|
c, x = layer(c, x, global_cond, **kwargs)
|
||||||
|
|
||||||
|
if len(self.single_layers) > 0:
|
||||||
|
c_len = c.size(1)
|
||||||
|
cx = torch.cat([c, x], dim=1)
|
||||||
|
for layer in self.single_layers:
|
||||||
|
cx = layer(cx, global_cond, **kwargs)
|
||||||
|
|
||||||
|
x = cx[:, c_len:]
|
||||||
|
|
||||||
|
fshift, fscale = self.modF(global_cond).chunk(2, dim=1)
|
||||||
|
|
||||||
|
x = modulate(x, fshift, fscale)
|
||||||
|
x = self.final_linear(x)
|
||||||
|
x = self.unpatchify(x, (h + 1) // self.patch_size, (w + 1) // self.patch_size)[:,:,:h,:w]
|
||||||
|
return x
|
||||||
@ -90,4 +90,4 @@ class ControlNet(nn.Module):
|
|||||||
proj_outputs = [None for _ in range(max(self.proj_blocks) + 1)]
|
proj_outputs = [None for _ in range(max(self.proj_blocks) + 1)]
|
||||||
for i, idx in enumerate(self.proj_blocks):
|
for i, idx in enumerate(self.proj_blocks):
|
||||||
proj_outputs[idx] = self.projections[i](x)
|
proj_outputs[idx] = self.projections[i](x)
|
||||||
return proj_outputs
|
return {"input": proj_outputs[::-1]}
|
||||||
|
|||||||
@ -17,7 +17,6 @@
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import math
|
import math
|
||||||
import numpy as np
|
|
||||||
import torch
|
import torch
|
||||||
from torch import nn
|
from torch import nn
|
||||||
from .common import AttnBlock, LayerNorm2d_op, ResBlock, FeedForwardBlock, TimestepBlock
|
from .common import AttnBlock, LayerNorm2d_op, ResBlock, FeedForwardBlock, TimestepBlock
|
||||||
|
|||||||
@ -18,7 +18,6 @@
|
|||||||
|
|
||||||
import torch
|
import torch
|
||||||
from torch import nn
|
from torch import nn
|
||||||
import numpy as np
|
|
||||||
import math
|
import math
|
||||||
from .common import AttnBlock, LayerNorm2d_op, ResBlock, FeedForwardBlock, TimestepBlock
|
from .common import AttnBlock, LayerNorm2d_op, ResBlock, FeedForwardBlock, TimestepBlock
|
||||||
# from .controlnet import ControlNetDeliverer
|
# from .controlnet import ControlNetDeliverer
|
||||||
|
|||||||
219
comfy/ldm/hydit/attn_layers.py
Normal file
219
comfy/ldm/hydit/attn_layers.py
Normal file
@ -0,0 +1,219 @@
|
|||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
from typing import Tuple, Union, Optional
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention
|
||||||
|
|
||||||
|
|
||||||
|
def reshape_for_broadcast(freqs_cis: Union[torch.Tensor, Tuple[torch.Tensor]], x: torch.Tensor, head_first=False):
|
||||||
|
"""
|
||||||
|
Reshape frequency tensor for broadcasting it with another tensor.
|
||||||
|
|
||||||
|
This function reshapes the frequency tensor to have the same shape as the target tensor 'x'
|
||||||
|
for the purpose of broadcasting the frequency tensor during element-wise operations.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
freqs_cis (Union[torch.Tensor, Tuple[torch.Tensor]]): Frequency tensor to be reshaped.
|
||||||
|
x (torch.Tensor): Target tensor for broadcasting compatibility.
|
||||||
|
head_first (bool): head dimension first (except batch dim) or not.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
torch.Tensor: Reshaped frequency tensor.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
AssertionError: If the frequency tensor doesn't match the expected shape.
|
||||||
|
AssertionError: If the target tensor 'x' doesn't have the expected number of dimensions.
|
||||||
|
"""
|
||||||
|
ndim = x.ndim
|
||||||
|
assert 0 <= 1 < ndim
|
||||||
|
|
||||||
|
if isinstance(freqs_cis, tuple):
|
||||||
|
# freqs_cis: (cos, sin) in real space
|
||||||
|
if head_first:
|
||||||
|
assert freqs_cis[0].shape == (x.shape[-2], x.shape[-1]), f'freqs_cis shape {freqs_cis[0].shape} does not match x shape {x.shape}'
|
||||||
|
shape = [d if i == ndim - 2 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
|
||||||
|
else:
|
||||||
|
assert freqs_cis[0].shape == (x.shape[1], x.shape[-1]), f'freqs_cis shape {freqs_cis[0].shape} does not match x shape {x.shape}'
|
||||||
|
shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
|
||||||
|
return freqs_cis[0].view(*shape), freqs_cis[1].view(*shape)
|
||||||
|
else:
|
||||||
|
# freqs_cis: values in complex space
|
||||||
|
if head_first:
|
||||||
|
assert freqs_cis.shape == (x.shape[-2], x.shape[-1]), f'freqs_cis shape {freqs_cis.shape} does not match x shape {x.shape}'
|
||||||
|
shape = [d if i == ndim - 2 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
|
||||||
|
else:
|
||||||
|
assert freqs_cis.shape == (x.shape[1], x.shape[-1]), f'freqs_cis shape {freqs_cis.shape} does not match x shape {x.shape}'
|
||||||
|
shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
|
||||||
|
return freqs_cis.view(*shape)
|
||||||
|
|
||||||
|
|
||||||
|
def rotate_half(x):
|
||||||
|
x_real, x_imag = x.float().reshape(*x.shape[:-1], -1, 2).unbind(-1) # [B, S, H, D//2]
|
||||||
|
return torch.stack([-x_imag, x_real], dim=-1).flatten(3)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_rotary_emb(
|
||||||
|
xq: torch.Tensor,
|
||||||
|
xk: Optional[torch.Tensor],
|
||||||
|
freqs_cis: Union[torch.Tensor, Tuple[torch.Tensor]],
|
||||||
|
head_first: bool = False,
|
||||||
|
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||||
|
"""
|
||||||
|
Apply rotary embeddings to input tensors using the given frequency tensor.
|
||||||
|
|
||||||
|
This function applies rotary embeddings to the given query 'xq' and key 'xk' tensors using the provided
|
||||||
|
frequency tensor 'freqs_cis'. The input tensors are reshaped as complex numbers, and the frequency tensor
|
||||||
|
is reshaped for broadcasting compatibility. The resulting tensors contain rotary embeddings and are
|
||||||
|
returned as real tensors.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
xq (torch.Tensor): Query tensor to apply rotary embeddings. [B, S, H, D]
|
||||||
|
xk (torch.Tensor): Key tensor to apply rotary embeddings. [B, S, H, D]
|
||||||
|
freqs_cis (Union[torch.Tensor, Tuple[torch.Tensor]]): Precomputed frequency tensor for complex exponentials.
|
||||||
|
head_first (bool): head dimension first (except batch dim) or not.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple[torch.Tensor, torch.Tensor]: Tuple of modified query tensor and key tensor with rotary embeddings.
|
||||||
|
|
||||||
|
"""
|
||||||
|
xk_out = None
|
||||||
|
if isinstance(freqs_cis, tuple):
|
||||||
|
cos, sin = reshape_for_broadcast(freqs_cis, xq, head_first) # [S, D]
|
||||||
|
cos, sin = cos.to(xq.device), sin.to(xq.device)
|
||||||
|
xq_out = (xq.float() * cos + rotate_half(xq.float()) * sin).type_as(xq)
|
||||||
|
if xk is not None:
|
||||||
|
xk_out = (xk.float() * cos + rotate_half(xk.float()) * sin).type_as(xk)
|
||||||
|
else:
|
||||||
|
xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2)) # [B, S, H, D//2]
|
||||||
|
freqs_cis = reshape_for_broadcast(freqs_cis, xq_, head_first).to(xq.device) # [S, D//2] --> [1, S, 1, D//2]
|
||||||
|
xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3).type_as(xq)
|
||||||
|
if xk is not None:
|
||||||
|
xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2)) # [B, S, H, D//2]
|
||||||
|
xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3).type_as(xk)
|
||||||
|
|
||||||
|
return xq_out, xk_out
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class CrossAttention(nn.Module):
|
||||||
|
"""
|
||||||
|
Use QK Normalization.
|
||||||
|
"""
|
||||||
|
def __init__(self,
|
||||||
|
qdim,
|
||||||
|
kdim,
|
||||||
|
num_heads,
|
||||||
|
qkv_bias=True,
|
||||||
|
qk_norm=False,
|
||||||
|
attn_drop=0.0,
|
||||||
|
proj_drop=0.0,
|
||||||
|
attn_precision=None,
|
||||||
|
device=None,
|
||||||
|
dtype=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
factory_kwargs = {'device': device, 'dtype': dtype}
|
||||||
|
super().__init__()
|
||||||
|
self.attn_precision = attn_precision
|
||||||
|
self.qdim = qdim
|
||||||
|
self.kdim = kdim
|
||||||
|
self.num_heads = num_heads
|
||||||
|
assert self.qdim % num_heads == 0, "self.qdim must be divisible by num_heads"
|
||||||
|
self.head_dim = self.qdim // num_heads
|
||||||
|
assert self.head_dim % 8 == 0 and self.head_dim <= 128, "Only support head_dim <= 128 and divisible by 8"
|
||||||
|
self.scale = self.head_dim ** -0.5
|
||||||
|
|
||||||
|
self.q_proj = operations.Linear(qdim, qdim, bias=qkv_bias, **factory_kwargs)
|
||||||
|
self.kv_proj = operations.Linear(kdim, 2 * qdim, bias=qkv_bias, **factory_kwargs)
|
||||||
|
|
||||||
|
# TODO: eps should be 1 / 65530 if using fp16
|
||||||
|
self.q_norm = operations.LayerNorm(self.head_dim, elementwise_affine=True, eps=1e-6, dtype=dtype, device=device) if qk_norm else nn.Identity()
|
||||||
|
self.k_norm = operations.LayerNorm(self.head_dim, elementwise_affine=True, eps=1e-6, dtype=dtype, device=device) if qk_norm else nn.Identity()
|
||||||
|
self.attn_drop = nn.Dropout(attn_drop)
|
||||||
|
self.out_proj = operations.Linear(qdim, qdim, bias=qkv_bias, **factory_kwargs)
|
||||||
|
self.proj_drop = nn.Dropout(proj_drop)
|
||||||
|
|
||||||
|
def forward(self, x, y, freqs_cis_img=None):
|
||||||
|
"""
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
x: torch.Tensor
|
||||||
|
(batch, seqlen1, hidden_dim) (where hidden_dim = num heads * head dim)
|
||||||
|
y: torch.Tensor
|
||||||
|
(batch, seqlen2, hidden_dim2)
|
||||||
|
freqs_cis_img: torch.Tensor
|
||||||
|
(batch, hidden_dim // 2), RoPE for image
|
||||||
|
"""
|
||||||
|
b, s1, c = x.shape # [b, s1, D]
|
||||||
|
_, s2, c = y.shape # [b, s2, 1024]
|
||||||
|
|
||||||
|
q = self.q_proj(x).view(b, s1, self.num_heads, self.head_dim) # [b, s1, h, d]
|
||||||
|
kv = self.kv_proj(y).view(b, s2, 2, self.num_heads, self.head_dim) # [b, s2, 2, h, d]
|
||||||
|
k, v = kv.unbind(dim=2) # [b, s, h, d]
|
||||||
|
q = self.q_norm(q)
|
||||||
|
k = self.k_norm(k)
|
||||||
|
|
||||||
|
# Apply RoPE if needed
|
||||||
|
if freqs_cis_img is not None:
|
||||||
|
qq, _ = apply_rotary_emb(q, None, freqs_cis_img)
|
||||||
|
assert qq.shape == q.shape, f'qq: {qq.shape}, q: {q.shape}'
|
||||||
|
q = qq
|
||||||
|
|
||||||
|
q = q.transpose(-2, -3).contiguous() # q -> B, L1, H, C - B, H, L1, C
|
||||||
|
k = k.transpose(-2, -3).contiguous() # k -> B, L2, H, C - B, H, C, L2
|
||||||
|
v = v.transpose(-2, -3).contiguous()
|
||||||
|
|
||||||
|
context = optimized_attention(q, k, v, self.num_heads, skip_reshape=True, attn_precision=self.attn_precision)
|
||||||
|
|
||||||
|
out = self.out_proj(context) # context.reshape - B, L1, -1
|
||||||
|
out = self.proj_drop(out)
|
||||||
|
|
||||||
|
out_tuple = (out,)
|
||||||
|
|
||||||
|
return out_tuple
|
||||||
|
|
||||||
|
|
||||||
|
class Attention(nn.Module):
|
||||||
|
"""
|
||||||
|
We rename some layer names to align with flash attention
|
||||||
|
"""
|
||||||
|
def __init__(self, dim, num_heads, qkv_bias=True, qk_norm=False, attn_drop=0., proj_drop=0., attn_precision=None, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.attn_precision = attn_precision
|
||||||
|
self.dim = dim
|
||||||
|
self.num_heads = num_heads
|
||||||
|
assert self.dim % num_heads == 0, 'dim should be divisible by num_heads'
|
||||||
|
self.head_dim = self.dim // num_heads
|
||||||
|
# This assertion is aligned with flash attention
|
||||||
|
assert self.head_dim % 8 == 0 and self.head_dim <= 128, "Only support head_dim <= 128 and divisible by 8"
|
||||||
|
self.scale = self.head_dim ** -0.5
|
||||||
|
|
||||||
|
# qkv --> Wqkv
|
||||||
|
self.Wqkv = operations.Linear(dim, dim * 3, bias=qkv_bias, dtype=dtype, device=device)
|
||||||
|
# TODO: eps should be 1 / 65530 if using fp16
|
||||||
|
self.q_norm = operations.LayerNorm(self.head_dim, elementwise_affine=True, eps=1e-6, dtype=dtype, device=device) if qk_norm else nn.Identity()
|
||||||
|
self.k_norm = operations.LayerNorm(self.head_dim, elementwise_affine=True, eps=1e-6, dtype=dtype, device=device) if qk_norm else nn.Identity()
|
||||||
|
self.attn_drop = nn.Dropout(attn_drop)
|
||||||
|
self.out_proj = operations.Linear(dim, dim, dtype=dtype, device=device)
|
||||||
|
self.proj_drop = nn.Dropout(proj_drop)
|
||||||
|
|
||||||
|
def forward(self, x, freqs_cis_img=None):
|
||||||
|
B, N, C = x.shape
|
||||||
|
qkv = self.Wqkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) # [3, b, h, s, d]
|
||||||
|
q, k, v = qkv.unbind(0) # [b, h, s, d]
|
||||||
|
q = self.q_norm(q) # [b, h, s, d]
|
||||||
|
k = self.k_norm(k) # [b, h, s, d]
|
||||||
|
|
||||||
|
# Apply RoPE if needed
|
||||||
|
if freqs_cis_img is not None:
|
||||||
|
qq, kk = apply_rotary_emb(q, k, freqs_cis_img, head_first=True)
|
||||||
|
assert qq.shape == q.shape and kk.shape == k.shape, \
|
||||||
|
f'qq: {qq.shape}, q: {q.shape}, kk: {kk.shape}, k: {k.shape}'
|
||||||
|
q, k = qq, kk
|
||||||
|
|
||||||
|
x = optimized_attention(q, k, v, self.num_heads, skip_reshape=True, attn_precision=self.attn_precision)
|
||||||
|
x = self.out_proj(x)
|
||||||
|
x = self.proj_drop(x)
|
||||||
|
|
||||||
|
out_tuple = (x,)
|
||||||
|
|
||||||
|
return out_tuple
|
||||||
404
comfy/ldm/hydit/models.py
Normal file
404
comfy/ldm/hydit/models.py
Normal file
@ -0,0 +1,404 @@
|
|||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
import torch.nn.functional as F
|
||||||
|
|
||||||
|
from comfy.ldm.modules.diffusionmodules.mmdit import Mlp, TimestepEmbedder, PatchEmbed, RMSNorm
|
||||||
|
from comfy.ldm.modules.diffusionmodules.util import timestep_embedding
|
||||||
|
from torch.utils import checkpoint
|
||||||
|
|
||||||
|
from .attn_layers import Attention, CrossAttention
|
||||||
|
from .poolers import AttentionPool
|
||||||
|
from .posemb_layers import get_2d_rotary_pos_embed, get_fill_resize_and_crop
|
||||||
|
|
||||||
|
def calc_rope(x, patch_size, head_size):
|
||||||
|
th = (x.shape[2] + (patch_size // 2)) // patch_size
|
||||||
|
tw = (x.shape[3] + (patch_size // 2)) // patch_size
|
||||||
|
base_size = 512 // 8 // patch_size
|
||||||
|
start, stop = get_fill_resize_and_crop((th, tw), base_size)
|
||||||
|
sub_args = [start, stop, (th, tw)]
|
||||||
|
# head_size = HUNYUAN_DIT_CONFIG['DiT-g/2']['hidden_size'] // HUNYUAN_DIT_CONFIG['DiT-g/2']['num_heads']
|
||||||
|
rope = get_2d_rotary_pos_embed(head_size, *sub_args)
|
||||||
|
return rope
|
||||||
|
|
||||||
|
|
||||||
|
def modulate(x, shift, scale):
|
||||||
|
return x * (1 + scale.unsqueeze(1)) + shift.unsqueeze(1)
|
||||||
|
|
||||||
|
|
||||||
|
class HunYuanDiTBlock(nn.Module):
|
||||||
|
"""
|
||||||
|
A HunYuanDiT block with `add` conditioning.
|
||||||
|
"""
|
||||||
|
def __init__(self,
|
||||||
|
hidden_size,
|
||||||
|
c_emb_size,
|
||||||
|
num_heads,
|
||||||
|
mlp_ratio=4.0,
|
||||||
|
text_states_dim=1024,
|
||||||
|
qk_norm=False,
|
||||||
|
norm_type="layer",
|
||||||
|
skip=False,
|
||||||
|
attn_precision=None,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
use_ele_affine = True
|
||||||
|
|
||||||
|
if norm_type == "layer":
|
||||||
|
norm_layer = operations.LayerNorm
|
||||||
|
elif norm_type == "rms":
|
||||||
|
norm_layer = RMSNorm
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown norm_type: {norm_type}")
|
||||||
|
|
||||||
|
# ========================= Self-Attention =========================
|
||||||
|
self.norm1 = norm_layer(hidden_size, elementwise_affine=use_ele_affine, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
self.attn1 = Attention(hidden_size, num_heads=num_heads, qkv_bias=True, qk_norm=qk_norm, attn_precision=attn_precision, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
# ========================= FFN =========================
|
||||||
|
self.norm2 = norm_layer(hidden_size, elementwise_affine=use_ele_affine, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
mlp_hidden_dim = int(hidden_size * mlp_ratio)
|
||||||
|
approx_gelu = lambda: nn.GELU(approximate="tanh")
|
||||||
|
self.mlp = Mlp(in_features=hidden_size, hidden_features=mlp_hidden_dim, act_layer=approx_gelu, drop=0, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
# ========================= Add =========================
|
||||||
|
# Simply use add like SDXL.
|
||||||
|
self.default_modulation = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(c_emb_size, hidden_size, bias=True, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
# ========================= Cross-Attention =========================
|
||||||
|
self.attn2 = CrossAttention(hidden_size, text_states_dim, num_heads=num_heads, qkv_bias=True,
|
||||||
|
qk_norm=qk_norm, attn_precision=attn_precision, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.norm3 = norm_layer(hidden_size, elementwise_affine=True, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
# ========================= Skip Connection =========================
|
||||||
|
if skip:
|
||||||
|
self.skip_norm = norm_layer(2 * hidden_size, elementwise_affine=True, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
self.skip_linear = operations.Linear(2 * hidden_size, hidden_size, dtype=dtype, device=device)
|
||||||
|
else:
|
||||||
|
self.skip_linear = None
|
||||||
|
|
||||||
|
self.gradient_checkpointing = False
|
||||||
|
|
||||||
|
def _forward(self, x, c=None, text_states=None, freq_cis_img=None, skip=None):
|
||||||
|
# Long Skip Connection
|
||||||
|
if self.skip_linear is not None:
|
||||||
|
cat = torch.cat([x, skip], dim=-1)
|
||||||
|
cat = self.skip_norm(cat)
|
||||||
|
x = self.skip_linear(cat)
|
||||||
|
|
||||||
|
# Self-Attention
|
||||||
|
shift_msa = self.default_modulation(c).unsqueeze(dim=1)
|
||||||
|
attn_inputs = (
|
||||||
|
self.norm1(x) + shift_msa, freq_cis_img,
|
||||||
|
)
|
||||||
|
x = x + self.attn1(*attn_inputs)[0]
|
||||||
|
|
||||||
|
# Cross-Attention
|
||||||
|
cross_inputs = (
|
||||||
|
self.norm3(x), text_states, freq_cis_img
|
||||||
|
)
|
||||||
|
x = x + self.attn2(*cross_inputs)[0]
|
||||||
|
|
||||||
|
# FFN Layer
|
||||||
|
mlp_inputs = self.norm2(x)
|
||||||
|
x = x + self.mlp(mlp_inputs)
|
||||||
|
|
||||||
|
return x
|
||||||
|
|
||||||
|
def forward(self, x, c=None, text_states=None, freq_cis_img=None, skip=None):
|
||||||
|
if self.gradient_checkpointing and self.training:
|
||||||
|
return checkpoint.checkpoint(self._forward, x, c, text_states, freq_cis_img, skip)
|
||||||
|
return self._forward(x, c, text_states, freq_cis_img, skip)
|
||||||
|
|
||||||
|
|
||||||
|
class FinalLayer(nn.Module):
|
||||||
|
"""
|
||||||
|
The final layer of HunYuanDiT.
|
||||||
|
"""
|
||||||
|
def __init__(self, final_hidden_size, c_emb_size, patch_size, out_channels, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.norm_final = operations.LayerNorm(final_hidden_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
self.linear = operations.Linear(final_hidden_size, patch_size * patch_size * out_channels, bias=True, dtype=dtype, device=device)
|
||||||
|
self.adaLN_modulation = nn.Sequential(
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(c_emb_size, 2 * final_hidden_size, bias=True, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x, c):
|
||||||
|
shift, scale = self.adaLN_modulation(c).chunk(2, dim=1)
|
||||||
|
x = modulate(self.norm_final(x), shift, scale)
|
||||||
|
x = self.linear(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class HunYuanDiT(nn.Module):
|
||||||
|
"""
|
||||||
|
HunYuanDiT: Diffusion model with a Transformer backbone.
|
||||||
|
|
||||||
|
Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers.
|
||||||
|
|
||||||
|
Inherit PeftAdapterMixin to be compatible with the PEFT training pipeline.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
args: argparse.Namespace
|
||||||
|
The arguments parsed by argparse.
|
||||||
|
input_size: tuple
|
||||||
|
The size of the input image.
|
||||||
|
patch_size: int
|
||||||
|
The size of the patch.
|
||||||
|
in_channels: int
|
||||||
|
The number of input channels.
|
||||||
|
hidden_size: int
|
||||||
|
The hidden size of the transformer backbone.
|
||||||
|
depth: int
|
||||||
|
The number of transformer blocks.
|
||||||
|
num_heads: int
|
||||||
|
The number of attention heads.
|
||||||
|
mlp_ratio: float
|
||||||
|
The ratio of the hidden size of the MLP in the transformer block.
|
||||||
|
log_fn: callable
|
||||||
|
The logging function.
|
||||||
|
"""
|
||||||
|
#@register_to_config
|
||||||
|
def __init__(self,
|
||||||
|
input_size: tuple = 32,
|
||||||
|
patch_size: int = 2,
|
||||||
|
in_channels: int = 4,
|
||||||
|
hidden_size: int = 1152,
|
||||||
|
depth: int = 28,
|
||||||
|
num_heads: int = 16,
|
||||||
|
mlp_ratio: float = 4.0,
|
||||||
|
text_states_dim = 1024,
|
||||||
|
text_states_dim_t5 = 2048,
|
||||||
|
text_len = 77,
|
||||||
|
text_len_t5 = 256,
|
||||||
|
qk_norm = True,# See http://arxiv.org/abs/2302.05442 for details.
|
||||||
|
size_cond = False,
|
||||||
|
use_style_cond = False,
|
||||||
|
learn_sigma = True,
|
||||||
|
norm = "layer",
|
||||||
|
log_fn: callable = print,
|
||||||
|
attn_precision=None,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.log_fn = log_fn
|
||||||
|
self.depth = depth
|
||||||
|
self.learn_sigma = learn_sigma
|
||||||
|
self.in_channels = in_channels
|
||||||
|
self.out_channels = in_channels * 2 if learn_sigma else in_channels
|
||||||
|
self.patch_size = patch_size
|
||||||
|
self.num_heads = num_heads
|
||||||
|
self.hidden_size = hidden_size
|
||||||
|
self.text_states_dim = text_states_dim
|
||||||
|
self.text_states_dim_t5 = text_states_dim_t5
|
||||||
|
self.text_len = text_len
|
||||||
|
self.text_len_t5 = text_len_t5
|
||||||
|
self.size_cond = size_cond
|
||||||
|
self.use_style_cond = use_style_cond
|
||||||
|
self.norm = norm
|
||||||
|
self.dtype = dtype
|
||||||
|
#import pdb
|
||||||
|
#pdb.set_trace()
|
||||||
|
|
||||||
|
self.mlp_t5 = nn.Sequential(
|
||||||
|
operations.Linear(self.text_states_dim_t5, self.text_states_dim_t5 * 4, bias=True, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(self.text_states_dim_t5 * 4, self.text_states_dim, bias=True, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
# learnable replace
|
||||||
|
self.text_embedding_padding = nn.Parameter(
|
||||||
|
torch.empty(self.text_len + self.text_len_t5, self.text_states_dim, dtype=dtype, device=device))
|
||||||
|
|
||||||
|
# Attention pooling
|
||||||
|
pooler_out_dim = 1024
|
||||||
|
self.pooler = AttentionPool(self.text_len_t5, self.text_states_dim_t5, num_heads=8, output_dim=pooler_out_dim, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
# Dimension of the extra input vectors
|
||||||
|
self.extra_in_dim = pooler_out_dim
|
||||||
|
|
||||||
|
if self.size_cond:
|
||||||
|
# Image size and crop size conditions
|
||||||
|
self.extra_in_dim += 6 * 256
|
||||||
|
|
||||||
|
if self.use_style_cond:
|
||||||
|
# Here we use a default learned embedder layer for future extension.
|
||||||
|
self.style_embedder = nn.Embedding(1, hidden_size, dtype=dtype, device=device)
|
||||||
|
self.extra_in_dim += hidden_size
|
||||||
|
|
||||||
|
# Text embedding for `add`
|
||||||
|
self.x_embedder = PatchEmbed(input_size, patch_size, in_channels, hidden_size, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.t_embedder = TimestepEmbedder(hidden_size, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.extra_embedder = nn.Sequential(
|
||||||
|
operations.Linear(self.extra_in_dim, hidden_size * 4, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(hidden_size * 4, hidden_size, bias=True, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Image embedding
|
||||||
|
num_patches = self.x_embedder.num_patches
|
||||||
|
|
||||||
|
# HUnYuanDiT Blocks
|
||||||
|
self.blocks = nn.ModuleList([
|
||||||
|
HunYuanDiTBlock(hidden_size=hidden_size,
|
||||||
|
c_emb_size=hidden_size,
|
||||||
|
num_heads=num_heads,
|
||||||
|
mlp_ratio=mlp_ratio,
|
||||||
|
text_states_dim=self.text_states_dim,
|
||||||
|
qk_norm=qk_norm,
|
||||||
|
norm_type=self.norm,
|
||||||
|
skip=layer > depth // 2,
|
||||||
|
attn_precision=attn_precision,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations,
|
||||||
|
)
|
||||||
|
for layer in range(depth)
|
||||||
|
])
|
||||||
|
|
||||||
|
self.final_layer = FinalLayer(hidden_size, hidden_size, patch_size, self.out_channels, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.unpatchify_channels = self.out_channels
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def forward(self,
|
||||||
|
x,
|
||||||
|
t,
|
||||||
|
context,#encoder_hidden_states=None,
|
||||||
|
text_embedding_mask=None,
|
||||||
|
encoder_hidden_states_t5=None,
|
||||||
|
text_embedding_mask_t5=None,
|
||||||
|
image_meta_size=None,
|
||||||
|
style=None,
|
||||||
|
return_dict=False,
|
||||||
|
control=None,
|
||||||
|
transformer_options=None,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Forward pass of the encoder.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
x: torch.Tensor
|
||||||
|
(B, D, H, W)
|
||||||
|
t: torch.Tensor
|
||||||
|
(B)
|
||||||
|
encoder_hidden_states: torch.Tensor
|
||||||
|
CLIP text embedding, (B, L_clip, D)
|
||||||
|
text_embedding_mask: torch.Tensor
|
||||||
|
CLIP text embedding mask, (B, L_clip)
|
||||||
|
encoder_hidden_states_t5: torch.Tensor
|
||||||
|
T5 text embedding, (B, L_t5, D)
|
||||||
|
text_embedding_mask_t5: torch.Tensor
|
||||||
|
T5 text embedding mask, (B, L_t5)
|
||||||
|
image_meta_size: torch.Tensor
|
||||||
|
(B, 6)
|
||||||
|
style: torch.Tensor
|
||||||
|
(B)
|
||||||
|
cos_cis_img: torch.Tensor
|
||||||
|
sin_cis_img: torch.Tensor
|
||||||
|
return_dict: bool
|
||||||
|
Whether to return a dictionary.
|
||||||
|
"""
|
||||||
|
#import pdb
|
||||||
|
#pdb.set_trace()
|
||||||
|
encoder_hidden_states = context
|
||||||
|
text_states = encoder_hidden_states # 2,77,1024
|
||||||
|
text_states_t5 = encoder_hidden_states_t5 # 2,256,2048
|
||||||
|
text_states_mask = text_embedding_mask.bool() # 2,77
|
||||||
|
text_states_t5_mask = text_embedding_mask_t5.bool() # 2,256
|
||||||
|
b_t5, l_t5, c_t5 = text_states_t5.shape
|
||||||
|
text_states_t5 = self.mlp_t5(text_states_t5.view(-1, c_t5)).view(b_t5, l_t5, -1)
|
||||||
|
|
||||||
|
padding = self.text_embedding_padding.to(text_states)
|
||||||
|
|
||||||
|
text_states[:,-self.text_len:] = torch.where(text_states_mask[:,-self.text_len:].unsqueeze(2), text_states[:,-self.text_len:], padding[:self.text_len])
|
||||||
|
text_states_t5[:,-self.text_len_t5:] = torch.where(text_states_t5_mask[:,-self.text_len_t5:].unsqueeze(2), text_states_t5[:,-self.text_len_t5:], padding[self.text_len:])
|
||||||
|
|
||||||
|
text_states = torch.cat([text_states, text_states_t5], dim=1) # 2,205,1024
|
||||||
|
# clip_t5_mask = torch.cat([text_states_mask, text_states_t5_mask], dim=-1)
|
||||||
|
|
||||||
|
_, _, oh, ow = x.shape
|
||||||
|
th, tw = (oh + (self.patch_size // 2)) // self.patch_size, (ow + (self.patch_size // 2)) // self.patch_size
|
||||||
|
|
||||||
|
|
||||||
|
# Get image RoPE embedding according to `reso`lution.
|
||||||
|
freqs_cis_img = calc_rope(x, self.patch_size, self.hidden_size // self.num_heads) #(cos_cis_img, sin_cis_img)
|
||||||
|
|
||||||
|
# ========================= Build time and image embedding =========================
|
||||||
|
t = self.t_embedder(t, dtype=x.dtype)
|
||||||
|
x = self.x_embedder(x)
|
||||||
|
|
||||||
|
# ========================= Concatenate all extra vectors =========================
|
||||||
|
# Build text tokens with pooling
|
||||||
|
extra_vec = self.pooler(encoder_hidden_states_t5)
|
||||||
|
|
||||||
|
# Build image meta size tokens if applicable
|
||||||
|
if self.size_cond:
|
||||||
|
image_meta_size = timestep_embedding(image_meta_size.view(-1), 256).to(x.dtype) # [B * 6, 256]
|
||||||
|
image_meta_size = image_meta_size.view(-1, 6 * 256)
|
||||||
|
extra_vec = torch.cat([extra_vec, image_meta_size], dim=1) # [B, D + 6 * 256]
|
||||||
|
|
||||||
|
# Build style tokens
|
||||||
|
if self.use_style_cond:
|
||||||
|
if style is None:
|
||||||
|
style = torch.zeros((extra_vec.shape[0],), device=x.device, dtype=torch.int)
|
||||||
|
style_embedding = self.style_embedder(style)
|
||||||
|
extra_vec = torch.cat([extra_vec, style_embedding], dim=1)
|
||||||
|
|
||||||
|
# Concatenate all extra vectors
|
||||||
|
c = t + self.extra_embedder(extra_vec) # [B, D]
|
||||||
|
|
||||||
|
controls = None
|
||||||
|
# ========================= Forward pass through HunYuanDiT blocks =========================
|
||||||
|
skips = []
|
||||||
|
for layer, block in enumerate(self.blocks):
|
||||||
|
if layer > self.depth // 2:
|
||||||
|
if controls is not None:
|
||||||
|
skip = skips.pop() + controls.pop()
|
||||||
|
else:
|
||||||
|
skip = skips.pop()
|
||||||
|
x = block(x, c, text_states, freqs_cis_img, skip) # (N, L, D)
|
||||||
|
else:
|
||||||
|
x = block(x, c, text_states, freqs_cis_img) # (N, L, D)
|
||||||
|
|
||||||
|
if layer < (self.depth // 2 - 1):
|
||||||
|
skips.append(x)
|
||||||
|
if controls is not None and len(controls) != 0:
|
||||||
|
raise ValueError("The number of controls is not equal to the number of skip connections.")
|
||||||
|
|
||||||
|
# ========================= Final layer =========================
|
||||||
|
x = self.final_layer(x, c) # (N, L, patch_size ** 2 * out_channels)
|
||||||
|
x = self.unpatchify(x, th, tw) # (N, out_channels, H, W)
|
||||||
|
|
||||||
|
if return_dict:
|
||||||
|
return {'x': x}
|
||||||
|
if self.learn_sigma:
|
||||||
|
return x[:,:self.out_channels // 2,:oh,:ow]
|
||||||
|
return x[:,:,:oh,:ow]
|
||||||
|
|
||||||
|
def unpatchify(self, x, h, w):
|
||||||
|
"""
|
||||||
|
x: (N, T, patch_size**2 * C)
|
||||||
|
imgs: (N, H, W, C)
|
||||||
|
"""
|
||||||
|
c = self.unpatchify_channels
|
||||||
|
p = self.x_embedder.patch_size[0]
|
||||||
|
# h = w = int(x.shape[1] ** 0.5)
|
||||||
|
assert h * w == x.shape[1]
|
||||||
|
|
||||||
|
x = x.reshape(shape=(x.shape[0], h, w, p, p, c))
|
||||||
|
x = torch.einsum('nhwpqc->nchpwq', x)
|
||||||
|
imgs = x.reshape(shape=(x.shape[0], c, h * p, w * p))
|
||||||
|
return imgs
|
||||||
37
comfy/ldm/hydit/poolers.py
Normal file
37
comfy/ldm/hydit/poolers.py
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
import torch.nn.functional as F
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention #TODO
|
||||||
|
|
||||||
|
|
||||||
|
class AttentionPool(nn.Module):
|
||||||
|
def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.positional_embedding = nn.Parameter(torch.empty(spacial_dim + 1, embed_dim, dtype=dtype, device=device))
|
||||||
|
self.k_proj = operations.Linear(embed_dim, embed_dim, dtype=dtype, device=device)
|
||||||
|
self.q_proj = operations.Linear(embed_dim, embed_dim, dtype=dtype, device=device)
|
||||||
|
self.v_proj = operations.Linear(embed_dim, embed_dim, dtype=dtype, device=device)
|
||||||
|
self.c_proj = operations.Linear(embed_dim, output_dim or embed_dim, dtype=dtype, device=device)
|
||||||
|
self.num_heads = num_heads
|
||||||
|
self.embed_dim = embed_dim
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = x[:,:self.positional_embedding.shape[0] - 1]
|
||||||
|
x = x.permute(1, 0, 2) # NLC -> LNC
|
||||||
|
x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0) # (L+1)NC
|
||||||
|
x = x + self.positional_embedding[:, None, :].to(dtype=x.dtype, device=x.device) # (L+1)NC
|
||||||
|
|
||||||
|
q = self.q_proj(x[:1])
|
||||||
|
k = self.k_proj(x)
|
||||||
|
v = self.v_proj(x)
|
||||||
|
|
||||||
|
batch_size = q.shape[1]
|
||||||
|
head_dim = self.embed_dim // self.num_heads
|
||||||
|
q = q.view(1, batch_size * self.num_heads, head_dim).transpose(0, 1).view(batch_size, self.num_heads, -1, head_dim)
|
||||||
|
k = k.view(k.shape[0], batch_size * self.num_heads, head_dim).transpose(0, 1).view(batch_size, self.num_heads, -1, head_dim)
|
||||||
|
v = v.view(v.shape[0], batch_size * self.num_heads, head_dim).transpose(0, 1).view(batch_size, self.num_heads, -1, head_dim)
|
||||||
|
|
||||||
|
attn_output = optimized_attention(q, k, v, self.num_heads, skip_reshape=True).transpose(0, 1)
|
||||||
|
|
||||||
|
attn_output = self.c_proj(attn_output)
|
||||||
|
return attn_output.squeeze(0)
|
||||||
224
comfy/ldm/hydit/posemb_layers.py
Normal file
224
comfy/ldm/hydit/posemb_layers.py
Normal file
@ -0,0 +1,224 @@
|
|||||||
|
import torch
|
||||||
|
import numpy as np
|
||||||
|
from typing import Union
|
||||||
|
|
||||||
|
|
||||||
|
def _to_tuple(x):
|
||||||
|
if isinstance(x, int):
|
||||||
|
return x, x
|
||||||
|
else:
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
def get_fill_resize_and_crop(src, tgt):
|
||||||
|
th, tw = _to_tuple(tgt)
|
||||||
|
h, w = _to_tuple(src)
|
||||||
|
|
||||||
|
tr = th / tw # base resolution
|
||||||
|
r = h / w # target resolution
|
||||||
|
|
||||||
|
# resize
|
||||||
|
if r > tr:
|
||||||
|
resize_height = th
|
||||||
|
resize_width = int(round(th / h * w))
|
||||||
|
else:
|
||||||
|
resize_width = tw
|
||||||
|
resize_height = int(round(tw / w * h)) # resize the target resolution down based on the base resolution
|
||||||
|
|
||||||
|
crop_top = int(round((th - resize_height) / 2.0))
|
||||||
|
crop_left = int(round((tw - resize_width) / 2.0))
|
||||||
|
|
||||||
|
return (crop_top, crop_left), (crop_top + resize_height, crop_left + resize_width)
|
||||||
|
|
||||||
|
|
||||||
|
def get_meshgrid(start, *args):
|
||||||
|
if len(args) == 0:
|
||||||
|
# start is grid_size
|
||||||
|
num = _to_tuple(start)
|
||||||
|
start = (0, 0)
|
||||||
|
stop = num
|
||||||
|
elif len(args) == 1:
|
||||||
|
# start is start, args[0] is stop, step is 1
|
||||||
|
start = _to_tuple(start)
|
||||||
|
stop = _to_tuple(args[0])
|
||||||
|
num = (stop[0] - start[0], stop[1] - start[1])
|
||||||
|
elif len(args) == 2:
|
||||||
|
# start is start, args[0] is stop, args[1] is num
|
||||||
|
start = _to_tuple(start)
|
||||||
|
stop = _to_tuple(args[0])
|
||||||
|
num = _to_tuple(args[1])
|
||||||
|
else:
|
||||||
|
raise ValueError(f"len(args) should be 0, 1 or 2, but got {len(args)}")
|
||||||
|
|
||||||
|
grid_h = np.linspace(start[0], stop[0], num[0], endpoint=False, dtype=np.float32)
|
||||||
|
grid_w = np.linspace(start[1], stop[1], num[1], endpoint=False, dtype=np.float32)
|
||||||
|
grid = np.meshgrid(grid_w, grid_h) # here w goes first
|
||||||
|
grid = np.stack(grid, axis=0) # [2, W, H]
|
||||||
|
return grid
|
||||||
|
|
||||||
|
#################################################################################
|
||||||
|
# Sine/Cosine Positional Embedding Functions #
|
||||||
|
#################################################################################
|
||||||
|
# https://github.com/facebookresearch/mae/blob/main/util/pos_embed.py
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed(embed_dim, start, *args, cls_token=False, extra_tokens=0):
|
||||||
|
"""
|
||||||
|
grid_size: int of the grid height and width
|
||||||
|
return:
|
||||||
|
pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token)
|
||||||
|
"""
|
||||||
|
grid = get_meshgrid(start, *args) # [2, H, w]
|
||||||
|
# grid_h = np.arange(grid_size, dtype=np.float32)
|
||||||
|
# grid_w = np.arange(grid_size, dtype=np.float32)
|
||||||
|
# grid = np.meshgrid(grid_w, grid_h) # here w goes first
|
||||||
|
# grid = np.stack(grid, axis=0) # [2, W, H]
|
||||||
|
|
||||||
|
grid = grid.reshape([2, 1, *grid.shape[1:]])
|
||||||
|
pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)
|
||||||
|
if cls_token and extra_tokens > 0:
|
||||||
|
pos_embed = np.concatenate([np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0)
|
||||||
|
return pos_embed
|
||||||
|
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
|
||||||
|
assert embed_dim % 2 == 0
|
||||||
|
|
||||||
|
# use half of dimensions to encode grid_h
|
||||||
|
emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2)
|
||||||
|
emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2)
|
||||||
|
|
||||||
|
emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
|
||||||
|
"""
|
||||||
|
embed_dim: output dimension for each position
|
||||||
|
pos: a list of positions to be encoded: size (W,H)
|
||||||
|
out: (M, D)
|
||||||
|
"""
|
||||||
|
assert embed_dim % 2 == 0
|
||||||
|
omega = np.arange(embed_dim // 2, dtype=np.float64)
|
||||||
|
omega /= embed_dim / 2.
|
||||||
|
omega = 1. / 10000**omega # (D/2,)
|
||||||
|
|
||||||
|
pos = pos.reshape(-1) # (M,)
|
||||||
|
out = np.einsum('m,d->md', pos, omega) # (M, D/2), outer product
|
||||||
|
|
||||||
|
emb_sin = np.sin(out) # (M, D/2)
|
||||||
|
emb_cos = np.cos(out) # (M, D/2)
|
||||||
|
|
||||||
|
emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
#################################################################################
|
||||||
|
# Rotary Positional Embedding Functions #
|
||||||
|
#################################################################################
|
||||||
|
# https://github.com/facebookresearch/llama/blob/main/llama/model.py#L443
|
||||||
|
|
||||||
|
def get_2d_rotary_pos_embed(embed_dim, start, *args, use_real=True):
|
||||||
|
"""
|
||||||
|
This is a 2d version of precompute_freqs_cis, which is a RoPE for image tokens with 2d structure.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
embed_dim: int
|
||||||
|
embedding dimension size
|
||||||
|
start: int or tuple of int
|
||||||
|
If len(args) == 0, start is num; If len(args) == 1, start is start, args[0] is stop, step is 1;
|
||||||
|
If len(args) == 2, start is start, args[0] is stop, args[1] is num.
|
||||||
|
use_real: bool
|
||||||
|
If True, return real part and imaginary part separately. Otherwise, return complex numbers.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
pos_embed: torch.Tensor
|
||||||
|
[HW, D/2]
|
||||||
|
"""
|
||||||
|
grid = get_meshgrid(start, *args) # [2, H, w]
|
||||||
|
grid = grid.reshape([2, 1, *grid.shape[1:]]) # Returns a sampling matrix with the same resolution as the target resolution
|
||||||
|
pos_embed = get_2d_rotary_pos_embed_from_grid(embed_dim, grid, use_real=use_real)
|
||||||
|
return pos_embed
|
||||||
|
|
||||||
|
|
||||||
|
def get_2d_rotary_pos_embed_from_grid(embed_dim, grid, use_real=False):
|
||||||
|
assert embed_dim % 4 == 0
|
||||||
|
|
||||||
|
# use half of dimensions to encode grid_h
|
||||||
|
emb_h = get_1d_rotary_pos_embed(embed_dim // 2, grid[0].reshape(-1), use_real=use_real) # (H*W, D/4)
|
||||||
|
emb_w = get_1d_rotary_pos_embed(embed_dim // 2, grid[1].reshape(-1), use_real=use_real) # (H*W, D/4)
|
||||||
|
|
||||||
|
if use_real:
|
||||||
|
cos = torch.cat([emb_h[0], emb_w[0]], dim=1) # (H*W, D/2)
|
||||||
|
sin = torch.cat([emb_h[1], emb_w[1]], dim=1) # (H*W, D/2)
|
||||||
|
return cos, sin
|
||||||
|
else:
|
||||||
|
emb = torch.cat([emb_h, emb_w], dim=1) # (H*W, D/2)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
def get_1d_rotary_pos_embed(dim: int, pos: Union[np.ndarray, int], theta: float = 10000.0, use_real=False):
|
||||||
|
"""
|
||||||
|
Precompute the frequency tensor for complex exponentials (cis) with given dimensions.
|
||||||
|
|
||||||
|
This function calculates a frequency tensor with complex exponentials using the given dimension 'dim'
|
||||||
|
and the end index 'end'. The 'theta' parameter scales the frequencies.
|
||||||
|
The returned tensor contains complex values in complex64 data type.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
dim (int): Dimension of the frequency tensor.
|
||||||
|
pos (np.ndarray, int): Position indices for the frequency tensor. [S] or scalar
|
||||||
|
theta (float, optional): Scaling factor for frequency computation. Defaults to 10000.0.
|
||||||
|
use_real (bool, optional): If True, return real part and imaginary part separately.
|
||||||
|
Otherwise, return complex numbers.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
torch.Tensor: Precomputed frequency tensor with complex exponentials. [S, D/2]
|
||||||
|
|
||||||
|
"""
|
||||||
|
if isinstance(pos, int):
|
||||||
|
pos = np.arange(pos)
|
||||||
|
freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim)) # [D/2]
|
||||||
|
t = torch.from_numpy(pos).to(freqs.device) # type: ignore # [S]
|
||||||
|
freqs = torch.outer(t, freqs).float() # type: ignore # [S, D/2]
|
||||||
|
if use_real:
|
||||||
|
freqs_cos = freqs.cos().repeat_interleave(2, dim=1) # [S, D]
|
||||||
|
freqs_sin = freqs.sin().repeat_interleave(2, dim=1) # [S, D]
|
||||||
|
return freqs_cos, freqs_sin
|
||||||
|
else:
|
||||||
|
freqs_cis = torch.polar(torch.ones_like(freqs), freqs) # complex64 # [S, D/2]
|
||||||
|
return freqs_cis
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def calc_sizes(rope_img, patch_size, th, tw):
|
||||||
|
if rope_img == 'extend':
|
||||||
|
# Expansion mode
|
||||||
|
sub_args = [(th, tw)]
|
||||||
|
elif rope_img.startswith('base'):
|
||||||
|
# Based on the specified dimensions, other dimensions are obtained through interpolation.
|
||||||
|
base_size = int(rope_img[4:]) // 8 // patch_size
|
||||||
|
start, stop = get_fill_resize_and_crop((th, tw), base_size)
|
||||||
|
sub_args = [start, stop, (th, tw)]
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown rope_img: {rope_img}")
|
||||||
|
return sub_args
|
||||||
|
|
||||||
|
|
||||||
|
def init_image_posemb(rope_img,
|
||||||
|
resolutions,
|
||||||
|
patch_size,
|
||||||
|
hidden_size,
|
||||||
|
num_heads,
|
||||||
|
log_fn,
|
||||||
|
rope_real=True,
|
||||||
|
):
|
||||||
|
freqs_cis_img = {}
|
||||||
|
for reso in resolutions:
|
||||||
|
th, tw = reso.height // 8 // patch_size, reso.width // 8 // patch_size
|
||||||
|
sub_args = calc_sizes(rope_img, patch_size, th, tw)
|
||||||
|
freqs_cis_img[str(reso)] = get_2d_rotary_pos_embed(hidden_size // num_heads, *sub_args, use_real=rope_real)
|
||||||
|
log_fn(f" Using image RoPE ({rope_img}) ({'real' if rope_real else 'complex'}): {sub_args} | ({reso}) "
|
||||||
|
f"{freqs_cis_img[str(reso)][0].shape if rope_real else freqs_cis_img[str(reso)].shape}")
|
||||||
|
return freqs_cis_img
|
||||||
@ -1,6 +1,4 @@
|
|||||||
import torch
|
import torch
|
||||||
# import pytorch_lightning as pl
|
|
||||||
import torch.nn.functional as F
|
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from typing import Any, Dict, List, Optional, Tuple, Union
|
from typing import Any, Dict, List, Optional, Tuple, Union
|
||||||
|
|
||||||
|
|||||||
@ -3,7 +3,7 @@ import torch
|
|||||||
import torch.nn.functional as F
|
import torch.nn.functional as F
|
||||||
from torch import nn, einsum
|
from torch import nn, einsum
|
||||||
from einops import rearrange, repeat
|
from einops import rearrange, repeat
|
||||||
from typing import Optional, Any
|
from typing import Optional
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
from .diffusionmodules.util import AlphaBlender, timestep_embedding
|
from .diffusionmodules.util import AlphaBlender, timestep_embedding
|
||||||
@ -86,22 +86,32 @@ class FeedForward(nn.Module):
|
|||||||
def Normalize(in_channels, dtype=None, device=None):
|
def Normalize(in_channels, dtype=None, device=None):
|
||||||
return torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True, dtype=dtype, device=device)
|
return torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True, dtype=dtype, device=device)
|
||||||
|
|
||||||
def attention_basic(q, k, v, heads, mask=None, attn_precision=None):
|
def attention_basic(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False):
|
||||||
attn_precision = get_attn_precision(attn_precision)
|
attn_precision = get_attn_precision(attn_precision)
|
||||||
|
|
||||||
b, _, dim_head = q.shape
|
if skip_reshape:
|
||||||
dim_head //= heads
|
b, _, _, dim_head = q.shape
|
||||||
|
else:
|
||||||
|
b, _, dim_head = q.shape
|
||||||
|
dim_head //= heads
|
||||||
|
|
||||||
scale = dim_head ** -0.5
|
scale = dim_head ** -0.5
|
||||||
|
|
||||||
h = heads
|
h = heads
|
||||||
q, k, v = map(
|
if skip_reshape:
|
||||||
lambda t: t.unsqueeze(3)
|
q, k, v = map(
|
||||||
.reshape(b, -1, heads, dim_head)
|
lambda t: t.reshape(b * heads, -1, dim_head),
|
||||||
.permute(0, 2, 1, 3)
|
(q, k, v),
|
||||||
.reshape(b * heads, -1, dim_head)
|
)
|
||||||
.contiguous(),
|
else:
|
||||||
(q, k, v),
|
q, k, v = map(
|
||||||
)
|
lambda t: t.unsqueeze(3)
|
||||||
|
.reshape(b, -1, heads, dim_head)
|
||||||
|
.permute(0, 2, 1, 3)
|
||||||
|
.reshape(b * heads, -1, dim_head)
|
||||||
|
.contiguous(),
|
||||||
|
(q, k, v),
|
||||||
|
)
|
||||||
|
|
||||||
# force cast to fp32 to avoid overflowing
|
# force cast to fp32 to avoid overflowing
|
||||||
if attn_precision == torch.float32:
|
if attn_precision == torch.float32:
|
||||||
@ -138,17 +148,26 @@ def attention_basic(q, k, v, heads, mask=None, attn_precision=None):
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def attention_sub_quad(query, key, value, heads, mask=None, attn_precision=None):
|
def attention_sub_quad(query, key, value, heads, mask=None, attn_precision=None, skip_reshape=False):
|
||||||
attn_precision = get_attn_precision(attn_precision)
|
attn_precision = get_attn_precision(attn_precision)
|
||||||
|
|
||||||
b, _, dim_head = query.shape
|
if skip_reshape:
|
||||||
dim_head //= heads
|
b, _, _, dim_head = query.shape
|
||||||
|
else:
|
||||||
|
b, _, dim_head = query.shape
|
||||||
|
dim_head //= heads
|
||||||
|
|
||||||
scale = dim_head ** -0.5
|
scale = dim_head ** -0.5
|
||||||
query = query.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
|
|
||||||
value = value.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
|
|
||||||
|
|
||||||
key = key.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 3, 1).reshape(b * heads, dim_head, -1)
|
if skip_reshape:
|
||||||
|
query = query.reshape(b * heads, -1, dim_head)
|
||||||
|
value = value.reshape(b * heads, -1, dim_head)
|
||||||
|
key = key.reshape(b * heads, -1, dim_head).movedim(1, 2)
|
||||||
|
else:
|
||||||
|
query = query.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
|
||||||
|
value = value.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 1, 3).reshape(b * heads, -1, dim_head)
|
||||||
|
key = key.unsqueeze(3).reshape(b, -1, heads, dim_head).permute(0, 2, 3, 1).reshape(b * heads, dim_head, -1)
|
||||||
|
|
||||||
|
|
||||||
dtype = query.dtype
|
dtype = query.dtype
|
||||||
upcast_attention = attn_precision == torch.float32 and query.dtype != torch.float32
|
upcast_attention = attn_precision == torch.float32 and query.dtype != torch.float32
|
||||||
@ -200,22 +219,32 @@ def attention_sub_quad(query, key, value, heads, mask=None, attn_precision=None)
|
|||||||
hidden_states = hidden_states.unflatten(0, (-1, heads)).transpose(1,2).flatten(start_dim=2)
|
hidden_states = hidden_states.unflatten(0, (-1, heads)).transpose(1,2).flatten(start_dim=2)
|
||||||
return hidden_states
|
return hidden_states
|
||||||
|
|
||||||
def attention_split(q, k, v, heads, mask=None, attn_precision=None):
|
def attention_split(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False):
|
||||||
attn_precision = get_attn_precision(attn_precision)
|
attn_precision = get_attn_precision(attn_precision)
|
||||||
|
|
||||||
b, _, dim_head = q.shape
|
if skip_reshape:
|
||||||
dim_head //= heads
|
b, _, _, dim_head = q.shape
|
||||||
|
else:
|
||||||
|
b, _, dim_head = q.shape
|
||||||
|
dim_head //= heads
|
||||||
|
|
||||||
scale = dim_head ** -0.5
|
scale = dim_head ** -0.5
|
||||||
|
|
||||||
h = heads
|
h = heads
|
||||||
q, k, v = map(
|
if skip_reshape:
|
||||||
lambda t: t.unsqueeze(3)
|
q, k, v = map(
|
||||||
.reshape(b, -1, heads, dim_head)
|
lambda t: t.reshape(b * heads, -1, dim_head),
|
||||||
.permute(0, 2, 1, 3)
|
(q, k, v),
|
||||||
.reshape(b * heads, -1, dim_head)
|
)
|
||||||
.contiguous(),
|
else:
|
||||||
(q, k, v),
|
q, k, v = map(
|
||||||
)
|
lambda t: t.unsqueeze(3)
|
||||||
|
.reshape(b, -1, heads, dim_head)
|
||||||
|
.permute(0, 2, 1, 3)
|
||||||
|
.reshape(b * heads, -1, dim_head)
|
||||||
|
.contiguous(),
|
||||||
|
(q, k, v),
|
||||||
|
)
|
||||||
|
|
||||||
r1 = torch.zeros(q.shape[0], q.shape[1], v.shape[2], device=q.device, dtype=q.dtype)
|
r1 = torch.zeros(q.shape[0], q.shape[1], v.shape[2], device=q.device, dtype=q.dtype)
|
||||||
|
|
||||||
@ -311,9 +340,12 @@ try:
|
|||||||
except:
|
except:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
def attention_xformers(q, k, v, heads, mask=None, attn_precision=None):
|
def attention_xformers(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False):
|
||||||
b, _, dim_head = q.shape
|
if skip_reshape:
|
||||||
dim_head //= heads
|
b, _, _, dim_head = q.shape
|
||||||
|
else:
|
||||||
|
b, _, dim_head = q.shape
|
||||||
|
dim_head //= heads
|
||||||
|
|
||||||
disabled_xformers = False
|
disabled_xformers = False
|
||||||
|
|
||||||
@ -328,10 +360,16 @@ def attention_xformers(q, k, v, heads, mask=None, attn_precision=None):
|
|||||||
if disabled_xformers:
|
if disabled_xformers:
|
||||||
return attention_pytorch(q, k, v, heads, mask)
|
return attention_pytorch(q, k, v, heads, mask)
|
||||||
|
|
||||||
q, k, v = map(
|
if skip_reshape:
|
||||||
lambda t: t.reshape(b, -1, heads, dim_head),
|
q, k, v = map(
|
||||||
(q, k, v),
|
lambda t: t.reshape(b * heads, -1, dim_head),
|
||||||
)
|
(q, k, v),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
q, k, v = map(
|
||||||
|
lambda t: t.reshape(b, -1, heads, dim_head),
|
||||||
|
(q, k, v),
|
||||||
|
)
|
||||||
|
|
||||||
if mask is not None:
|
if mask is not None:
|
||||||
pad = 8 - q.shape[1] % 8
|
pad = 8 - q.shape[1] % 8
|
||||||
@ -341,18 +379,30 @@ def attention_xformers(q, k, v, heads, mask=None, attn_precision=None):
|
|||||||
|
|
||||||
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)
|
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)
|
||||||
|
|
||||||
out = (
|
if skip_reshape:
|
||||||
out.reshape(b, -1, heads * dim_head)
|
out = (
|
||||||
)
|
out.unsqueeze(0)
|
||||||
|
.reshape(b, heads, -1, dim_head)
|
||||||
|
.permute(0, 2, 1, 3)
|
||||||
|
.reshape(b, -1, heads * dim_head)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
out = (
|
||||||
|
out.reshape(b, -1, heads * dim_head)
|
||||||
|
)
|
||||||
|
|
||||||
return out
|
return out
|
||||||
|
|
||||||
def attention_pytorch(q, k, v, heads, mask=None, attn_precision=None):
|
def attention_pytorch(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False):
|
||||||
b, _, dim_head = q.shape
|
if skip_reshape:
|
||||||
dim_head //= heads
|
b, _, _, dim_head = q.shape
|
||||||
q, k, v = map(
|
else:
|
||||||
lambda t: t.view(b, -1, heads, dim_head).transpose(1, 2),
|
b, _, dim_head = q.shape
|
||||||
(q, k, v),
|
dim_head //= heads
|
||||||
)
|
q, k, v = map(
|
||||||
|
lambda t: t.view(b, -1, heads, dim_head).transpose(1, 2),
|
||||||
|
(q, k, v),
|
||||||
|
)
|
||||||
|
|
||||||
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
|
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
|
||||||
out = (
|
out = (
|
||||||
|
|||||||
953
comfy/ldm/modules/diffusionmodules/mmdit.py
Normal file
953
comfy/ldm/modules/diffusionmodules/mmdit.py
Normal file
@ -0,0 +1,953 @@
|
|||||||
|
import logging
|
||||||
|
import math
|
||||||
|
from typing import Dict, Optional
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
from .. import attention
|
||||||
|
from einops import rearrange, repeat
|
||||||
|
from .util import timestep_embedding
|
||||||
|
|
||||||
|
def default(x, y):
|
||||||
|
if x is not None:
|
||||||
|
return x
|
||||||
|
return y
|
||||||
|
|
||||||
|
class Mlp(nn.Module):
|
||||||
|
""" MLP as used in Vision Transformer, MLP-Mixer and related networks
|
||||||
|
"""
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
in_features,
|
||||||
|
hidden_features=None,
|
||||||
|
out_features=None,
|
||||||
|
act_layer=nn.GELU,
|
||||||
|
norm_layer=None,
|
||||||
|
bias=True,
|
||||||
|
drop=0.,
|
||||||
|
use_conv=False,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
out_features = out_features or in_features
|
||||||
|
hidden_features = hidden_features or in_features
|
||||||
|
drop_probs = drop
|
||||||
|
linear_layer = partial(operations.Conv2d, kernel_size=1) if use_conv else operations.Linear
|
||||||
|
|
||||||
|
self.fc1 = linear_layer(in_features, hidden_features, bias=bias, dtype=dtype, device=device)
|
||||||
|
self.act = act_layer()
|
||||||
|
self.drop1 = nn.Dropout(drop_probs)
|
||||||
|
self.norm = norm_layer(hidden_features) if norm_layer is not None else nn.Identity()
|
||||||
|
self.fc2 = linear_layer(hidden_features, out_features, bias=bias, dtype=dtype, device=device)
|
||||||
|
self.drop2 = nn.Dropout(drop_probs)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = self.fc1(x)
|
||||||
|
x = self.act(x)
|
||||||
|
x = self.drop1(x)
|
||||||
|
x = self.norm(x)
|
||||||
|
x = self.fc2(x)
|
||||||
|
x = self.drop2(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
class PatchEmbed(nn.Module):
|
||||||
|
""" 2D Image to Patch Embedding
|
||||||
|
"""
|
||||||
|
dynamic_img_pad: torch.jit.Final[bool]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
img_size: Optional[int] = 224,
|
||||||
|
patch_size: int = 16,
|
||||||
|
in_chans: int = 3,
|
||||||
|
embed_dim: int = 768,
|
||||||
|
norm_layer = None,
|
||||||
|
flatten: bool = True,
|
||||||
|
bias: bool = True,
|
||||||
|
strict_img_size: bool = True,
|
||||||
|
dynamic_img_pad: bool = True,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.patch_size = (patch_size, patch_size)
|
||||||
|
if img_size is not None:
|
||||||
|
self.img_size = (img_size, img_size)
|
||||||
|
self.grid_size = tuple([s // p for s, p in zip(self.img_size, self.patch_size)])
|
||||||
|
self.num_patches = self.grid_size[0] * self.grid_size[1]
|
||||||
|
else:
|
||||||
|
self.img_size = None
|
||||||
|
self.grid_size = None
|
||||||
|
self.num_patches = None
|
||||||
|
|
||||||
|
# flatten spatial dim and transpose to channels last, kept for bwd compat
|
||||||
|
self.flatten = flatten
|
||||||
|
self.strict_img_size = strict_img_size
|
||||||
|
self.dynamic_img_pad = dynamic_img_pad
|
||||||
|
|
||||||
|
self.proj = operations.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size, bias=bias, dtype=dtype, device=device)
|
||||||
|
self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
B, C, H, W = x.shape
|
||||||
|
# if self.img_size is not None:
|
||||||
|
# if self.strict_img_size:
|
||||||
|
# _assert(H == self.img_size[0], f"Input height ({H}) doesn't match model ({self.img_size[0]}).")
|
||||||
|
# _assert(W == self.img_size[1], f"Input width ({W}) doesn't match model ({self.img_size[1]}).")
|
||||||
|
# elif not self.dynamic_img_pad:
|
||||||
|
# _assert(
|
||||||
|
# H % self.patch_size[0] == 0,
|
||||||
|
# f"Input height ({H}) should be divisible by patch size ({self.patch_size[0]})."
|
||||||
|
# )
|
||||||
|
# _assert(
|
||||||
|
# W % self.patch_size[1] == 0,
|
||||||
|
# f"Input width ({W}) should be divisible by patch size ({self.patch_size[1]})."
|
||||||
|
# )
|
||||||
|
if self.dynamic_img_pad:
|
||||||
|
pad_h = (self.patch_size[0] - H % self.patch_size[0]) % self.patch_size[0]
|
||||||
|
pad_w = (self.patch_size[1] - W % self.patch_size[1]) % self.patch_size[1]
|
||||||
|
x = torch.nn.functional.pad(x, (0, pad_w, 0, pad_h), mode='reflect')
|
||||||
|
x = self.proj(x)
|
||||||
|
if self.flatten:
|
||||||
|
x = x.flatten(2).transpose(1, 2) # NCHW -> NLC
|
||||||
|
x = self.norm(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
def modulate(x, shift, scale):
|
||||||
|
if shift is None:
|
||||||
|
shift = torch.zeros_like(scale)
|
||||||
|
return x * (1 + scale.unsqueeze(1)) + shift.unsqueeze(1)
|
||||||
|
|
||||||
|
|
||||||
|
#################################################################################
|
||||||
|
# Sine/Cosine Positional Embedding Functions #
|
||||||
|
#################################################################################
|
||||||
|
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed(
|
||||||
|
embed_dim,
|
||||||
|
grid_size,
|
||||||
|
cls_token=False,
|
||||||
|
extra_tokens=0,
|
||||||
|
scaling_factor=None,
|
||||||
|
offset=None,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
grid_size: int of the grid height and width
|
||||||
|
return:
|
||||||
|
pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token)
|
||||||
|
"""
|
||||||
|
grid_h = np.arange(grid_size, dtype=np.float32)
|
||||||
|
grid_w = np.arange(grid_size, dtype=np.float32)
|
||||||
|
grid = np.meshgrid(grid_w, grid_h) # here w goes first
|
||||||
|
grid = np.stack(grid, axis=0)
|
||||||
|
if scaling_factor is not None:
|
||||||
|
grid = grid / scaling_factor
|
||||||
|
if offset is not None:
|
||||||
|
grid = grid - offset
|
||||||
|
|
||||||
|
grid = grid.reshape([2, 1, grid_size, grid_size])
|
||||||
|
pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)
|
||||||
|
if cls_token and extra_tokens > 0:
|
||||||
|
pos_embed = np.concatenate(
|
||||||
|
[np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0
|
||||||
|
)
|
||||||
|
return pos_embed
|
||||||
|
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
|
||||||
|
assert embed_dim % 2 == 0
|
||||||
|
|
||||||
|
# use half of dimensions to encode grid_h
|
||||||
|
emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2)
|
||||||
|
emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2)
|
||||||
|
|
||||||
|
emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
|
||||||
|
"""
|
||||||
|
embed_dim: output dimension for each position
|
||||||
|
pos: a list of positions to be encoded: size (M,)
|
||||||
|
out: (M, D)
|
||||||
|
"""
|
||||||
|
assert embed_dim % 2 == 0
|
||||||
|
omega = np.arange(embed_dim // 2, dtype=np.float64)
|
||||||
|
omega /= embed_dim / 2.0
|
||||||
|
omega = 1.0 / 10000**omega # (D/2,)
|
||||||
|
|
||||||
|
pos = pos.reshape(-1) # (M,)
|
||||||
|
out = np.einsum("m,d->md", pos, omega) # (M, D/2), outer product
|
||||||
|
|
||||||
|
emb_sin = np.sin(out) # (M, D/2)
|
||||||
|
emb_cos = np.cos(out) # (M, D/2)
|
||||||
|
|
||||||
|
emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
def get_1d_sincos_pos_embed_from_grid_torch(embed_dim, pos, device=None, dtype=torch.float32):
|
||||||
|
omega = torch.arange(embed_dim // 2, device=device, dtype=dtype)
|
||||||
|
omega /= embed_dim / 2.0
|
||||||
|
omega = 1.0 / 10000**omega # (D/2,)
|
||||||
|
pos = pos.reshape(-1) # (M,)
|
||||||
|
out = torch.einsum("m,d->md", pos, omega) # (M, D/2), outer product
|
||||||
|
emb_sin = torch.sin(out) # (M, D/2)
|
||||||
|
emb_cos = torch.cos(out) # (M, D/2)
|
||||||
|
emb = torch.cat([emb_sin, emb_cos], dim=1) # (M, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed_torch(embed_dim, w, h, val_center=7.5, val_magnitude=7.5, device=None, dtype=torch.float32):
|
||||||
|
small = min(h, w)
|
||||||
|
val_h = (h / small) * val_magnitude
|
||||||
|
val_w = (w / small) * val_magnitude
|
||||||
|
grid_h, grid_w = torch.meshgrid(torch.linspace(-val_h + val_center, val_h + val_center, h, device=device, dtype=dtype), torch.linspace(-val_w + val_center, val_w + val_center, w, device=device, dtype=dtype), indexing='ij')
|
||||||
|
emb_h = get_1d_sincos_pos_embed_from_grid_torch(embed_dim // 2, grid_h, device=device, dtype=dtype)
|
||||||
|
emb_w = get_1d_sincos_pos_embed_from_grid_torch(embed_dim // 2, grid_w, device=device, dtype=dtype)
|
||||||
|
emb = torch.cat([emb_w, emb_h], dim=1) # (H*W, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
#################################################################################
|
||||||
|
# Embedding Layers for Timesteps and Class Labels #
|
||||||
|
#################################################################################
|
||||||
|
|
||||||
|
|
||||||
|
class TimestepEmbedder(nn.Module):
|
||||||
|
"""
|
||||||
|
Embeds scalar timesteps into vector representations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, hidden_size, frequency_embedding_size=256, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.mlp = nn.Sequential(
|
||||||
|
operations.Linear(frequency_embedding_size, hidden_size, bias=True, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(hidden_size, hidden_size, bias=True, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
self.frequency_embedding_size = frequency_embedding_size
|
||||||
|
|
||||||
|
def forward(self, t, dtype, **kwargs):
|
||||||
|
t_freq = timestep_embedding(t, self.frequency_embedding_size).to(dtype)
|
||||||
|
t_emb = self.mlp(t_freq)
|
||||||
|
return t_emb
|
||||||
|
|
||||||
|
|
||||||
|
class VectorEmbedder(nn.Module):
|
||||||
|
"""
|
||||||
|
Embeds a flat vector of dimension input_dim
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, input_dim: int, hidden_size: int, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.mlp = nn.Sequential(
|
||||||
|
operations.Linear(input_dim, hidden_size, bias=True, dtype=dtype, device=device),
|
||||||
|
nn.SiLU(),
|
||||||
|
operations.Linear(hidden_size, hidden_size, bias=True, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
emb = self.mlp(x)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
#################################################################################
|
||||||
|
# Core DiT Model #
|
||||||
|
#################################################################################
|
||||||
|
|
||||||
|
|
||||||
|
def split_qkv(qkv, head_dim):
|
||||||
|
qkv = qkv.reshape(qkv.shape[0], qkv.shape[1], 3, -1, head_dim).movedim(2, 0)
|
||||||
|
return qkv[0], qkv[1], qkv[2]
|
||||||
|
|
||||||
|
def optimized_attention(qkv, num_heads):
|
||||||
|
return attention.optimized_attention(qkv[0], qkv[1], qkv[2], num_heads)
|
||||||
|
|
||||||
|
class SelfAttention(nn.Module):
|
||||||
|
ATTENTION_MODES = ("xformers", "torch", "torch-hb", "math", "debug")
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim: int,
|
||||||
|
num_heads: int = 8,
|
||||||
|
qkv_bias: bool = False,
|
||||||
|
qk_scale: Optional[float] = None,
|
||||||
|
proj_drop: float = 0.0,
|
||||||
|
attn_mode: str = "xformers",
|
||||||
|
pre_only: bool = False,
|
||||||
|
qk_norm: Optional[str] = None,
|
||||||
|
rmsnorm: bool = False,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.num_heads = num_heads
|
||||||
|
self.head_dim = dim // num_heads
|
||||||
|
|
||||||
|
self.qkv = operations.Linear(dim, dim * 3, bias=qkv_bias, dtype=dtype, device=device)
|
||||||
|
if not pre_only:
|
||||||
|
self.proj = operations.Linear(dim, dim, dtype=dtype, device=device)
|
||||||
|
self.proj_drop = nn.Dropout(proj_drop)
|
||||||
|
assert attn_mode in self.ATTENTION_MODES
|
||||||
|
self.attn_mode = attn_mode
|
||||||
|
self.pre_only = pre_only
|
||||||
|
|
||||||
|
if qk_norm == "rms":
|
||||||
|
self.ln_q = RMSNorm(self.head_dim, elementwise_affine=True, eps=1.0e-6, dtype=dtype, device=device)
|
||||||
|
self.ln_k = RMSNorm(self.head_dim, elementwise_affine=True, eps=1.0e-6, dtype=dtype, device=device)
|
||||||
|
elif qk_norm == "ln":
|
||||||
|
self.ln_q = operations.LayerNorm(self.head_dim, elementwise_affine=True, eps=1.0e-6, dtype=dtype, device=device)
|
||||||
|
self.ln_k = operations.LayerNorm(self.head_dim, elementwise_affine=True, eps=1.0e-6, dtype=dtype, device=device)
|
||||||
|
elif qk_norm is None:
|
||||||
|
self.ln_q = nn.Identity()
|
||||||
|
self.ln_k = nn.Identity()
|
||||||
|
else:
|
||||||
|
raise ValueError(qk_norm)
|
||||||
|
|
||||||
|
def pre_attention(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
B, L, C = x.shape
|
||||||
|
qkv = self.qkv(x)
|
||||||
|
q, k, v = split_qkv(qkv, self.head_dim)
|
||||||
|
q = self.ln_q(q).reshape(q.shape[0], q.shape[1], -1)
|
||||||
|
k = self.ln_k(k).reshape(q.shape[0], q.shape[1], -1)
|
||||||
|
return (q, k, v)
|
||||||
|
|
||||||
|
def post_attention(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
assert not self.pre_only
|
||||||
|
x = self.proj(x)
|
||||||
|
x = self.proj_drop(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
qkv = self.pre_attention(x)
|
||||||
|
x = optimized_attention(
|
||||||
|
qkv, num_heads=self.num_heads
|
||||||
|
)
|
||||||
|
x = self.post_attention(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class RMSNorm(torch.nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self, dim: int, elementwise_affine: bool = False, eps: float = 1e-6, device=None, dtype=None
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Initialize the RMSNorm normalization layer.
|
||||||
|
Args:
|
||||||
|
dim (int): The dimension of the input tensor.
|
||||||
|
eps (float, optional): A small value added to the denominator for numerical stability. Default is 1e-6.
|
||||||
|
Attributes:
|
||||||
|
eps (float): A small value added to the denominator for numerical stability.
|
||||||
|
weight (nn.Parameter): Learnable scaling parameter.
|
||||||
|
"""
|
||||||
|
super().__init__()
|
||||||
|
self.eps = eps
|
||||||
|
self.learnable_scale = elementwise_affine
|
||||||
|
if self.learnable_scale:
|
||||||
|
self.weight = nn.Parameter(torch.empty(dim, device=device, dtype=dtype))
|
||||||
|
else:
|
||||||
|
self.register_parameter("weight", None)
|
||||||
|
|
||||||
|
def _norm(self, x):
|
||||||
|
"""
|
||||||
|
Apply the RMSNorm normalization to the input tensor.
|
||||||
|
Args:
|
||||||
|
x (torch.Tensor): The input tensor.
|
||||||
|
Returns:
|
||||||
|
torch.Tensor: The normalized tensor.
|
||||||
|
"""
|
||||||
|
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
"""
|
||||||
|
Forward pass through the RMSNorm layer.
|
||||||
|
Args:
|
||||||
|
x (torch.Tensor): The input tensor.
|
||||||
|
Returns:
|
||||||
|
torch.Tensor: The output tensor after applying RMSNorm.
|
||||||
|
"""
|
||||||
|
x = self._norm(x)
|
||||||
|
if self.learnable_scale:
|
||||||
|
return x * self.weight.to(device=x.device, dtype=x.dtype)
|
||||||
|
else:
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class SwiGLUFeedForward(nn.Module):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
dim: int,
|
||||||
|
hidden_dim: int,
|
||||||
|
multiple_of: int,
|
||||||
|
ffn_dim_multiplier: Optional[float] = None,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Initialize the FeedForward module.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
dim (int): Input dimension.
|
||||||
|
hidden_dim (int): Hidden dimension of the feedforward layer.
|
||||||
|
multiple_of (int): Value to ensure hidden dimension is a multiple of this value.
|
||||||
|
ffn_dim_multiplier (float, optional): Custom multiplier for hidden dimension. Defaults to None.
|
||||||
|
|
||||||
|
Attributes:
|
||||||
|
w1 (ColumnParallelLinear): Linear transformation for the first layer.
|
||||||
|
w2 (RowParallelLinear): Linear transformation for the second layer.
|
||||||
|
w3 (ColumnParallelLinear): Linear transformation for the third layer.
|
||||||
|
|
||||||
|
"""
|
||||||
|
super().__init__()
|
||||||
|
hidden_dim = int(2 * hidden_dim / 3)
|
||||||
|
# custom dim factor multiplier
|
||||||
|
if ffn_dim_multiplier is not None:
|
||||||
|
hidden_dim = int(ffn_dim_multiplier * hidden_dim)
|
||||||
|
hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)
|
||||||
|
|
||||||
|
self.w1 = nn.Linear(dim, hidden_dim, bias=False)
|
||||||
|
self.w2 = nn.Linear(hidden_dim, dim, bias=False)
|
||||||
|
self.w3 = nn.Linear(dim, hidden_dim, bias=False)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
return self.w2(nn.functional.silu(self.w1(x)) * self.w3(x))
|
||||||
|
|
||||||
|
|
||||||
|
class DismantledBlock(nn.Module):
|
||||||
|
"""
|
||||||
|
A DiT block with gated adaptive layer norm (adaLN) conditioning.
|
||||||
|
"""
|
||||||
|
|
||||||
|
ATTENTION_MODES = ("xformers", "torch", "torch-hb", "math", "debug")
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
hidden_size: int,
|
||||||
|
num_heads: int,
|
||||||
|
mlp_ratio: float = 4.0,
|
||||||
|
attn_mode: str = "xformers",
|
||||||
|
qkv_bias: bool = False,
|
||||||
|
pre_only: bool = False,
|
||||||
|
rmsnorm: bool = False,
|
||||||
|
scale_mod_only: bool = False,
|
||||||
|
swiglu: bool = False,
|
||||||
|
qk_norm: Optional[str] = None,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
**block_kwargs,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
assert attn_mode in self.ATTENTION_MODES
|
||||||
|
if not rmsnorm:
|
||||||
|
self.norm1 = operations.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
else:
|
||||||
|
self.norm1 = RMSNorm(hidden_size, elementwise_affine=False, eps=1e-6)
|
||||||
|
self.attn = SelfAttention(
|
||||||
|
dim=hidden_size,
|
||||||
|
num_heads=num_heads,
|
||||||
|
qkv_bias=qkv_bias,
|
||||||
|
attn_mode=attn_mode,
|
||||||
|
pre_only=pre_only,
|
||||||
|
qk_norm=qk_norm,
|
||||||
|
rmsnorm=rmsnorm,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations
|
||||||
|
)
|
||||||
|
if not pre_only:
|
||||||
|
if not rmsnorm:
|
||||||
|
self.norm2 = operations.LayerNorm(
|
||||||
|
hidden_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.norm2 = RMSNorm(hidden_size, elementwise_affine=False, eps=1e-6)
|
||||||
|
mlp_hidden_dim = int(hidden_size * mlp_ratio)
|
||||||
|
if not pre_only:
|
||||||
|
if not swiglu:
|
||||||
|
self.mlp = Mlp(
|
||||||
|
in_features=hidden_size,
|
||||||
|
hidden_features=mlp_hidden_dim,
|
||||||
|
act_layer=lambda: nn.GELU(approximate="tanh"),
|
||||||
|
drop=0,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.mlp = SwiGLUFeedForward(
|
||||||
|
dim=hidden_size,
|
||||||
|
hidden_dim=mlp_hidden_dim,
|
||||||
|
multiple_of=256,
|
||||||
|
)
|
||||||
|
self.scale_mod_only = scale_mod_only
|
||||||
|
if not scale_mod_only:
|
||||||
|
n_mods = 6 if not pre_only else 2
|
||||||
|
else:
|
||||||
|
n_mods = 4 if not pre_only else 1
|
||||||
|
self.adaLN_modulation = nn.Sequential(
|
||||||
|
nn.SiLU(), operations.Linear(hidden_size, n_mods * hidden_size, bias=True, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
self.pre_only = pre_only
|
||||||
|
|
||||||
|
def pre_attention(self, x: torch.Tensor, c: torch.Tensor) -> torch.Tensor:
|
||||||
|
if not self.pre_only:
|
||||||
|
if not self.scale_mod_only:
|
||||||
|
(
|
||||||
|
shift_msa,
|
||||||
|
scale_msa,
|
||||||
|
gate_msa,
|
||||||
|
shift_mlp,
|
||||||
|
scale_mlp,
|
||||||
|
gate_mlp,
|
||||||
|
) = self.adaLN_modulation(c).chunk(6, dim=1)
|
||||||
|
else:
|
||||||
|
shift_msa = None
|
||||||
|
shift_mlp = None
|
||||||
|
(
|
||||||
|
scale_msa,
|
||||||
|
gate_msa,
|
||||||
|
scale_mlp,
|
||||||
|
gate_mlp,
|
||||||
|
) = self.adaLN_modulation(
|
||||||
|
c
|
||||||
|
).chunk(4, dim=1)
|
||||||
|
qkv = self.attn.pre_attention(modulate(self.norm1(x), shift_msa, scale_msa))
|
||||||
|
return qkv, (
|
||||||
|
x,
|
||||||
|
gate_msa,
|
||||||
|
shift_mlp,
|
||||||
|
scale_mlp,
|
||||||
|
gate_mlp,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
if not self.scale_mod_only:
|
||||||
|
(
|
||||||
|
shift_msa,
|
||||||
|
scale_msa,
|
||||||
|
) = self.adaLN_modulation(
|
||||||
|
c
|
||||||
|
).chunk(2, dim=1)
|
||||||
|
else:
|
||||||
|
shift_msa = None
|
||||||
|
scale_msa = self.adaLN_modulation(c)
|
||||||
|
qkv = self.attn.pre_attention(modulate(self.norm1(x), shift_msa, scale_msa))
|
||||||
|
return qkv, None
|
||||||
|
|
||||||
|
def post_attention(self, attn, x, gate_msa, shift_mlp, scale_mlp, gate_mlp):
|
||||||
|
assert not self.pre_only
|
||||||
|
x = x + gate_msa.unsqueeze(1) * self.attn.post_attention(attn)
|
||||||
|
x = x + gate_mlp.unsqueeze(1) * self.mlp(
|
||||||
|
modulate(self.norm2(x), shift_mlp, scale_mlp)
|
||||||
|
)
|
||||||
|
return x
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor, c: torch.Tensor) -> torch.Tensor:
|
||||||
|
assert not self.pre_only
|
||||||
|
qkv, intermediates = self.pre_attention(x, c)
|
||||||
|
attn = optimized_attention(
|
||||||
|
qkv,
|
||||||
|
num_heads=self.attn.num_heads,
|
||||||
|
)
|
||||||
|
return self.post_attention(attn, *intermediates)
|
||||||
|
|
||||||
|
|
||||||
|
def block_mixing(*args, use_checkpoint=True, **kwargs):
|
||||||
|
if use_checkpoint:
|
||||||
|
return torch.utils.checkpoint.checkpoint(
|
||||||
|
_block_mixing, *args, use_reentrant=False, **kwargs
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return _block_mixing(*args, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def _block_mixing(context, x, context_block, x_block, c):
|
||||||
|
context_qkv, context_intermediates = context_block.pre_attention(context, c)
|
||||||
|
|
||||||
|
x_qkv, x_intermediates = x_block.pre_attention(x, c)
|
||||||
|
|
||||||
|
o = []
|
||||||
|
for t in range(3):
|
||||||
|
o.append(torch.cat((context_qkv[t], x_qkv[t]), dim=1))
|
||||||
|
qkv = tuple(o)
|
||||||
|
|
||||||
|
attn = optimized_attention(
|
||||||
|
qkv,
|
||||||
|
num_heads=x_block.attn.num_heads,
|
||||||
|
)
|
||||||
|
context_attn, x_attn = (
|
||||||
|
attn[:, : context_qkv[0].shape[1]],
|
||||||
|
attn[:, context_qkv[0].shape[1] :],
|
||||||
|
)
|
||||||
|
|
||||||
|
if not context_block.pre_only:
|
||||||
|
context = context_block.post_attention(context_attn, *context_intermediates)
|
||||||
|
|
||||||
|
else:
|
||||||
|
context = None
|
||||||
|
x = x_block.post_attention(x_attn, *x_intermediates)
|
||||||
|
return context, x
|
||||||
|
|
||||||
|
|
||||||
|
class JointBlock(nn.Module):
|
||||||
|
"""just a small wrapper to serve as a fsdp unit"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
*args,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
pre_only = kwargs.pop("pre_only")
|
||||||
|
qk_norm = kwargs.pop("qk_norm", None)
|
||||||
|
self.context_block = DismantledBlock(*args, pre_only=pre_only, qk_norm=qk_norm, **kwargs)
|
||||||
|
self.x_block = DismantledBlock(*args, pre_only=False, qk_norm=qk_norm, **kwargs)
|
||||||
|
|
||||||
|
def forward(self, *args, **kwargs):
|
||||||
|
return block_mixing(
|
||||||
|
*args, context_block=self.context_block, x_block=self.x_block, **kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FinalLayer(nn.Module):
|
||||||
|
"""
|
||||||
|
The final layer of DiT.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
hidden_size: int,
|
||||||
|
patch_size: int,
|
||||||
|
out_channels: int,
|
||||||
|
total_out_channels: Optional[int] = None,
|
||||||
|
dtype=None,
|
||||||
|
device=None,
|
||||||
|
operations=None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.norm_final = operations.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
self.linear = (
|
||||||
|
operations.Linear(hidden_size, patch_size * patch_size * out_channels, bias=True, dtype=dtype, device=device)
|
||||||
|
if (total_out_channels is None)
|
||||||
|
else operations.Linear(hidden_size, total_out_channels, bias=True, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
self.adaLN_modulation = nn.Sequential(
|
||||||
|
nn.SiLU(), operations.Linear(hidden_size, 2 * hidden_size, bias=True, dtype=dtype, device=device)
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor, c: torch.Tensor) -> torch.Tensor:
|
||||||
|
shift, scale = self.adaLN_modulation(c).chunk(2, dim=1)
|
||||||
|
x = modulate(self.norm_final(x), shift, scale)
|
||||||
|
x = self.linear(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
class SelfAttentionContext(nn.Module):
|
||||||
|
def __init__(self, dim, heads=8, dim_head=64, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
dim_head = dim // heads
|
||||||
|
inner_dim = dim
|
||||||
|
|
||||||
|
self.heads = heads
|
||||||
|
self.dim_head = dim_head
|
||||||
|
|
||||||
|
self.qkv = operations.Linear(dim, dim * 3, bias=True, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.proj = operations.Linear(inner_dim, dim, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
qkv = self.qkv(x)
|
||||||
|
q, k, v = split_qkv(qkv, self.dim_head)
|
||||||
|
x = optimized_attention((q.reshape(q.shape[0], q.shape[1], -1), k, v), self.heads)
|
||||||
|
return self.proj(x)
|
||||||
|
|
||||||
|
class ContextProcessorBlock(nn.Module):
|
||||||
|
def __init__(self, context_size, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.norm1 = operations.LayerNorm(context_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
self.attn = SelfAttentionContext(context_size, dtype=dtype, device=device, operations=operations)
|
||||||
|
self.norm2 = operations.LayerNorm(context_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
self.mlp = Mlp(in_features=context_size, hidden_features=(context_size * 4), act_layer=lambda: nn.GELU(approximate="tanh"), drop=0, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x += self.attn(self.norm1(x))
|
||||||
|
x += self.mlp(self.norm2(x))
|
||||||
|
return x
|
||||||
|
|
||||||
|
class ContextProcessor(nn.Module):
|
||||||
|
def __init__(self, context_size, num_layers, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.layers = torch.nn.ModuleList([ContextProcessorBlock(context_size, dtype=dtype, device=device, operations=operations) for i in range(num_layers)])
|
||||||
|
self.norm = operations.LayerNorm(context_size, elementwise_affine=False, eps=1e-6, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
for i, l in enumerate(self.layers):
|
||||||
|
x = l(x)
|
||||||
|
return self.norm(x)
|
||||||
|
|
||||||
|
class MMDiT(nn.Module):
|
||||||
|
"""
|
||||||
|
Diffusion model with a Transformer backbone.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
input_size: int = 32,
|
||||||
|
patch_size: int = 2,
|
||||||
|
in_channels: int = 4,
|
||||||
|
depth: int = 28,
|
||||||
|
# hidden_size: Optional[int] = None,
|
||||||
|
# num_heads: Optional[int] = None,
|
||||||
|
mlp_ratio: float = 4.0,
|
||||||
|
learn_sigma: bool = False,
|
||||||
|
adm_in_channels: Optional[int] = None,
|
||||||
|
context_embedder_config: Optional[Dict] = None,
|
||||||
|
compile_core: bool = False,
|
||||||
|
use_checkpoint: bool = False,
|
||||||
|
register_length: int = 0,
|
||||||
|
attn_mode: str = "torch",
|
||||||
|
rmsnorm: bool = False,
|
||||||
|
scale_mod_only: bool = False,
|
||||||
|
swiglu: bool = False,
|
||||||
|
out_channels: Optional[int] = None,
|
||||||
|
pos_embed_scaling_factor: Optional[float] = None,
|
||||||
|
pos_embed_offset: Optional[float] = None,
|
||||||
|
pos_embed_max_size: Optional[int] = None,
|
||||||
|
num_patches = None,
|
||||||
|
qk_norm: Optional[str] = None,
|
||||||
|
qkv_bias: bool = True,
|
||||||
|
context_processor_layers = None,
|
||||||
|
context_size = 4096,
|
||||||
|
num_blocks = None,
|
||||||
|
final_layer = True,
|
||||||
|
dtype = None, #TODO
|
||||||
|
device = None,
|
||||||
|
operations = None,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.dtype = dtype
|
||||||
|
self.learn_sigma = learn_sigma
|
||||||
|
self.in_channels = in_channels
|
||||||
|
default_out_channels = in_channels * 2 if learn_sigma else in_channels
|
||||||
|
self.out_channels = default(out_channels, default_out_channels)
|
||||||
|
self.patch_size = patch_size
|
||||||
|
self.pos_embed_scaling_factor = pos_embed_scaling_factor
|
||||||
|
self.pos_embed_offset = pos_embed_offset
|
||||||
|
self.pos_embed_max_size = pos_embed_max_size
|
||||||
|
|
||||||
|
# hidden_size = default(hidden_size, 64 * depth)
|
||||||
|
# num_heads = default(num_heads, hidden_size // 64)
|
||||||
|
|
||||||
|
# apply magic --> this defines a head_size of 64
|
||||||
|
self.hidden_size = 64 * depth
|
||||||
|
num_heads = depth
|
||||||
|
if num_blocks is None:
|
||||||
|
num_blocks = depth
|
||||||
|
|
||||||
|
self.depth = depth
|
||||||
|
self.num_heads = num_heads
|
||||||
|
|
||||||
|
self.x_embedder = PatchEmbed(
|
||||||
|
input_size,
|
||||||
|
patch_size,
|
||||||
|
in_channels,
|
||||||
|
self.hidden_size,
|
||||||
|
bias=True,
|
||||||
|
strict_img_size=self.pos_embed_max_size is None,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations
|
||||||
|
)
|
||||||
|
self.t_embedder = TimestepEmbedder(self.hidden_size, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
self.y_embedder = None
|
||||||
|
if adm_in_channels is not None:
|
||||||
|
assert isinstance(adm_in_channels, int)
|
||||||
|
self.y_embedder = VectorEmbedder(adm_in_channels, self.hidden_size, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
if context_processor_layers is not None:
|
||||||
|
self.context_processor = ContextProcessor(context_size, context_processor_layers, dtype=dtype, device=device, operations=operations)
|
||||||
|
else:
|
||||||
|
self.context_processor = None
|
||||||
|
|
||||||
|
self.context_embedder = nn.Identity()
|
||||||
|
if context_embedder_config is not None:
|
||||||
|
if context_embedder_config["target"] == "torch.nn.Linear":
|
||||||
|
self.context_embedder = operations.Linear(**context_embedder_config["params"], dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.register_length = register_length
|
||||||
|
if self.register_length > 0:
|
||||||
|
self.register = nn.Parameter(torch.randn(1, register_length, self.hidden_size, dtype=dtype, device=device))
|
||||||
|
|
||||||
|
# num_patches = self.x_embedder.num_patches
|
||||||
|
# Will use fixed sin-cos embedding:
|
||||||
|
# just use a buffer already
|
||||||
|
if num_patches is not None:
|
||||||
|
self.register_buffer(
|
||||||
|
"pos_embed",
|
||||||
|
torch.empty(1, num_patches, self.hidden_size, dtype=dtype, device=device),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.pos_embed = None
|
||||||
|
|
||||||
|
self.use_checkpoint = use_checkpoint
|
||||||
|
self.joint_blocks = nn.ModuleList(
|
||||||
|
[
|
||||||
|
JointBlock(
|
||||||
|
self.hidden_size,
|
||||||
|
num_heads,
|
||||||
|
mlp_ratio=mlp_ratio,
|
||||||
|
qkv_bias=qkv_bias,
|
||||||
|
attn_mode=attn_mode,
|
||||||
|
pre_only=(i == num_blocks - 1) and final_layer,
|
||||||
|
rmsnorm=rmsnorm,
|
||||||
|
scale_mod_only=scale_mod_only,
|
||||||
|
swiglu=swiglu,
|
||||||
|
qk_norm=qk_norm,
|
||||||
|
dtype=dtype,
|
||||||
|
device=device,
|
||||||
|
operations=operations
|
||||||
|
)
|
||||||
|
for i in range(num_blocks)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
if final_layer:
|
||||||
|
self.final_layer = FinalLayer(self.hidden_size, patch_size, self.out_channels, dtype=dtype, device=device, operations=operations)
|
||||||
|
|
||||||
|
if compile_core:
|
||||||
|
assert False
|
||||||
|
self.forward_core_with_concat = torch.compile(self.forward_core_with_concat)
|
||||||
|
|
||||||
|
def cropped_pos_embed(self, hw, device=None):
|
||||||
|
p = self.x_embedder.patch_size[0]
|
||||||
|
h, w = hw
|
||||||
|
# patched size
|
||||||
|
h = (h + 1) // p
|
||||||
|
w = (w + 1) // p
|
||||||
|
if self.pos_embed is None:
|
||||||
|
return get_2d_sincos_pos_embed_torch(self.hidden_size, w, h, device=device)
|
||||||
|
assert self.pos_embed_max_size is not None
|
||||||
|
assert h <= self.pos_embed_max_size, (h, self.pos_embed_max_size)
|
||||||
|
assert w <= self.pos_embed_max_size, (w, self.pos_embed_max_size)
|
||||||
|
top = (self.pos_embed_max_size - h) // 2
|
||||||
|
left = (self.pos_embed_max_size - w) // 2
|
||||||
|
spatial_pos_embed = rearrange(
|
||||||
|
self.pos_embed,
|
||||||
|
"1 (h w) c -> 1 h w c",
|
||||||
|
h=self.pos_embed_max_size,
|
||||||
|
w=self.pos_embed_max_size,
|
||||||
|
)
|
||||||
|
spatial_pos_embed = spatial_pos_embed[:, top : top + h, left : left + w, :]
|
||||||
|
spatial_pos_embed = rearrange(spatial_pos_embed, "1 h w c -> 1 (h w) c")
|
||||||
|
# print(spatial_pos_embed, top, left, h, w)
|
||||||
|
# # t = get_2d_sincos_pos_embed_torch(self.hidden_size, w, h, 7.875, 7.875, device=device) #matches exactly for 1024 res
|
||||||
|
# t = get_2d_sincos_pos_embed_torch(self.hidden_size, w, h, 7.5, 7.5, device=device) #scales better
|
||||||
|
# # print(t)
|
||||||
|
# return t
|
||||||
|
return spatial_pos_embed
|
||||||
|
|
||||||
|
def unpatchify(self, x, hw=None):
|
||||||
|
"""
|
||||||
|
x: (N, T, patch_size**2 * C)
|
||||||
|
imgs: (N, H, W, C)
|
||||||
|
"""
|
||||||
|
c = self.out_channels
|
||||||
|
p = self.x_embedder.patch_size[0]
|
||||||
|
if hw is None:
|
||||||
|
h = w = int(x.shape[1] ** 0.5)
|
||||||
|
else:
|
||||||
|
h, w = hw
|
||||||
|
h = (h + 1) // p
|
||||||
|
w = (w + 1) // p
|
||||||
|
assert h * w == x.shape[1]
|
||||||
|
|
||||||
|
x = x.reshape(shape=(x.shape[0], h, w, p, p, c))
|
||||||
|
x = torch.einsum("nhwpqc->nchpwq", x)
|
||||||
|
imgs = x.reshape(shape=(x.shape[0], c, h * p, w * p))
|
||||||
|
return imgs
|
||||||
|
|
||||||
|
def forward_core_with_concat(
|
||||||
|
self,
|
||||||
|
x: torch.Tensor,
|
||||||
|
c_mod: torch.Tensor,
|
||||||
|
context: Optional[torch.Tensor] = None,
|
||||||
|
control = None,
|
||||||
|
) -> torch.Tensor:
|
||||||
|
if self.register_length > 0:
|
||||||
|
context = torch.cat(
|
||||||
|
(
|
||||||
|
repeat(self.register, "1 ... -> b ...", b=x.shape[0]),
|
||||||
|
default(context, torch.Tensor([]).type_as(x)),
|
||||||
|
),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
# context is B, L', D
|
||||||
|
# x is B, L, D
|
||||||
|
blocks = len(self.joint_blocks)
|
||||||
|
for i in range(blocks):
|
||||||
|
context, x = self.joint_blocks[i](
|
||||||
|
context,
|
||||||
|
x,
|
||||||
|
c=c_mod,
|
||||||
|
use_checkpoint=self.use_checkpoint,
|
||||||
|
)
|
||||||
|
if control is not None:
|
||||||
|
control_o = control.get("output")
|
||||||
|
if i < len(control_o):
|
||||||
|
add = control_o[i]
|
||||||
|
if add is not None:
|
||||||
|
x += add
|
||||||
|
|
||||||
|
x = self.final_layer(x, c_mod) # (N, T, patch_size ** 2 * out_channels)
|
||||||
|
return x
|
||||||
|
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x: torch.Tensor,
|
||||||
|
t: torch.Tensor,
|
||||||
|
y: Optional[torch.Tensor] = None,
|
||||||
|
context: Optional[torch.Tensor] = None,
|
||||||
|
control = None,
|
||||||
|
) -> torch.Tensor:
|
||||||
|
"""
|
||||||
|
Forward pass of DiT.
|
||||||
|
x: (N, C, H, W) tensor of spatial inputs (images or latent representations of images)
|
||||||
|
t: (N,) tensor of diffusion timesteps
|
||||||
|
y: (N,) tensor of class labels
|
||||||
|
"""
|
||||||
|
|
||||||
|
if self.context_processor is not None:
|
||||||
|
context = self.context_processor(context)
|
||||||
|
|
||||||
|
hw = x.shape[-2:]
|
||||||
|
x = self.x_embedder(x) + self.cropped_pos_embed(hw, device=x.device).to(dtype=x.dtype, device=x.device)
|
||||||
|
c = self.t_embedder(t, dtype=x.dtype) # (N, D)
|
||||||
|
if y is not None and self.y_embedder is not None:
|
||||||
|
y = self.y_embedder(y) # (N, D)
|
||||||
|
c = c + y # (N, D)
|
||||||
|
|
||||||
|
if context is not None:
|
||||||
|
context = self.context_embedder(context)
|
||||||
|
|
||||||
|
x = self.forward_core_with_concat(x, c, context, control)
|
||||||
|
|
||||||
|
x = self.unpatchify(x, hw=hw) # (N, out_channels, H, W)
|
||||||
|
return x[:,:,:hw[-2],:hw[-1]]
|
||||||
|
|
||||||
|
|
||||||
|
class OpenAISignatureMMDITWrapper(MMDiT):
|
||||||
|
def forward(
|
||||||
|
self,
|
||||||
|
x: torch.Tensor,
|
||||||
|
timesteps: torch.Tensor,
|
||||||
|
context: Optional[torch.Tensor] = None,
|
||||||
|
y: Optional[torch.Tensor] = None,
|
||||||
|
control = None,
|
||||||
|
**kwargs,
|
||||||
|
) -> torch.Tensor:
|
||||||
|
return super().forward(x, timesteps, context=context, y=y, control=control)
|
||||||
|
|
||||||
@ -29,6 +29,8 @@ def load_lora(lora, to_load):
|
|||||||
|
|
||||||
regular_lora = "{}.lora_up.weight".format(x)
|
regular_lora = "{}.lora_up.weight".format(x)
|
||||||
diffusers_lora = "{}_lora.up.weight".format(x)
|
diffusers_lora = "{}_lora.up.weight".format(x)
|
||||||
|
diffusers2_lora = "{}.lora_B.weight".format(x)
|
||||||
|
diffusers3_lora = "{}.lora.up.weight".format(x)
|
||||||
transformers_lora = "{}.lora_linear_layer.up.weight".format(x)
|
transformers_lora = "{}.lora_linear_layer.up.weight".format(x)
|
||||||
A_name = None
|
A_name = None
|
||||||
|
|
||||||
@ -40,6 +42,14 @@ def load_lora(lora, to_load):
|
|||||||
A_name = diffusers_lora
|
A_name = diffusers_lora
|
||||||
B_name = "{}_lora.down.weight".format(x)
|
B_name = "{}_lora.down.weight".format(x)
|
||||||
mid_name = None
|
mid_name = None
|
||||||
|
elif diffusers2_lora in lora.keys():
|
||||||
|
A_name = diffusers2_lora
|
||||||
|
B_name = "{}.lora_A.weight".format(x)
|
||||||
|
mid_name = None
|
||||||
|
elif diffusers3_lora in lora.keys():
|
||||||
|
A_name = diffusers3_lora
|
||||||
|
B_name = "{}.lora.down.weight".format(x)
|
||||||
|
mid_name = None
|
||||||
elif transformers_lora in lora.keys():
|
elif transformers_lora in lora.keys():
|
||||||
A_name = transformers_lora
|
A_name = transformers_lora
|
||||||
B_name ="{}.lora_linear_layer.down.weight".format(x)
|
B_name ="{}.lora_linear_layer.down.weight".format(x)
|
||||||
@ -164,6 +174,7 @@ def load_lora(lora, to_load):
|
|||||||
for x in lora.keys():
|
for x in lora.keys():
|
||||||
if x not in loaded_keys:
|
if x not in loaded_keys:
|
||||||
logging.warning("lora key not loaded: {}".format(x))
|
logging.warning("lora key not loaded: {}".format(x))
|
||||||
|
|
||||||
return patch_dict
|
return patch_dict
|
||||||
|
|
||||||
def model_lora_keys_clip(model, key_map={}):
|
def model_lora_keys_clip(model, key_map={}):
|
||||||
@ -207,17 +218,27 @@ def model_lora_keys_clip(model, key_map={}):
|
|||||||
lora_key = "lora_prior_te_text_model_encoder_layers_{}_{}".format(b, LORA_CLIP_MAP[c]) #cascade lora: TODO put lora key prefix in the model config
|
lora_key = "lora_prior_te_text_model_encoder_layers_{}_{}".format(b, LORA_CLIP_MAP[c]) #cascade lora: TODO put lora key prefix in the model config
|
||||||
key_map[lora_key] = k
|
key_map[lora_key] = k
|
||||||
|
|
||||||
|
for k in sdk: #OneTrainer SD3 lora
|
||||||
|
if k.startswith("t5xxl.transformer.") and k.endswith(".weight"):
|
||||||
|
l_key = k[len("t5xxl.transformer."):-len(".weight")]
|
||||||
|
lora_key = "lora_te3_{}".format(l_key.replace(".", "_"))
|
||||||
|
key_map[lora_key] = k
|
||||||
|
|
||||||
k = "clip_g.transformer.text_projection.weight"
|
k = "clip_g.transformer.text_projection.weight"
|
||||||
if k in sdk:
|
if k in sdk:
|
||||||
key_map["lora_prior_te_text_projection"] = k #cascade lora?
|
key_map["lora_prior_te_text_projection"] = k #cascade lora?
|
||||||
# key_map["text_encoder.text_projection"] = k #TODO: check if other lora have the text_projection too
|
# key_map["text_encoder.text_projection"] = k #TODO: check if other lora have the text_projection too
|
||||||
# key_map["lora_te_text_projection"] = k
|
key_map["lora_te2_text_projection"] = k #OneTrainer SD3 lora
|
||||||
|
|
||||||
|
k = "clip_l.transformer.text_projection.weight"
|
||||||
|
if k in sdk:
|
||||||
|
key_map["lora_te1_text_projection"] = k #OneTrainer SD3 lora, not necessary but omits warning
|
||||||
|
|
||||||
return key_map
|
return key_map
|
||||||
|
|
||||||
def model_lora_keys_unet(model, key_map={}):
|
def model_lora_keys_unet(model, key_map={}):
|
||||||
sdk = model.state_dict().keys()
|
sd = model.state_dict()
|
||||||
|
sdk = sd.keys()
|
||||||
|
|
||||||
for k in sdk:
|
for k in sdk:
|
||||||
if k.startswith("diffusion_model.") and k.endswith(".weight"):
|
if k.startswith("diffusion_model.") and k.endswith(".weight"):
|
||||||
@ -238,4 +259,33 @@ def model_lora_keys_unet(model, key_map={}):
|
|||||||
if diffusers_lora_key.endswith(".to_out.0"):
|
if diffusers_lora_key.endswith(".to_out.0"):
|
||||||
diffusers_lora_key = diffusers_lora_key[:-2]
|
diffusers_lora_key = diffusers_lora_key[:-2]
|
||||||
key_map[diffusers_lora_key] = unet_key
|
key_map[diffusers_lora_key] = unet_key
|
||||||
|
|
||||||
|
if isinstance(model, comfy.model_base.SD3): #Diffusers lora SD3
|
||||||
|
diffusers_keys = comfy.utils.mmdit_to_diffusers(model.model_config.unet_config, output_prefix="diffusion_model.")
|
||||||
|
for k in diffusers_keys:
|
||||||
|
if k.endswith(".weight"):
|
||||||
|
to = diffusers_keys[k]
|
||||||
|
key_lora = "transformer.{}".format(k[:-len(".weight")]) #regular diffusers sd3 lora format
|
||||||
|
key_map[key_lora] = to
|
||||||
|
|
||||||
|
key_lora = "base_model.model.{}".format(k[:-len(".weight")]) #format for flash-sd3 lora and others?
|
||||||
|
key_map[key_lora] = to
|
||||||
|
|
||||||
|
key_lora = "lora_transformer_{}".format(k[:-len(".weight")].replace(".", "_")) #OneTrainer lora
|
||||||
|
key_map[key_lora] = to
|
||||||
|
|
||||||
|
if isinstance(model, comfy.model_base.AuraFlow): #Diffusers lora AuraFlow
|
||||||
|
diffusers_keys = comfy.utils.auraflow_to_diffusers(model.model_config.unet_config, output_prefix="diffusion_model.")
|
||||||
|
for k in diffusers_keys:
|
||||||
|
if k.endswith(".weight"):
|
||||||
|
to = diffusers_keys[k]
|
||||||
|
key_lora = "transformer.{}".format(k[:-len(".weight")]) #simpletrainer and probably regular diffusers lora format
|
||||||
|
key_map[key_lora] = to
|
||||||
|
|
||||||
|
if isinstance(model, comfy.model_base.HunyuanDiT):
|
||||||
|
for k in sdk:
|
||||||
|
if k.startswith("diffusion_model.") and k.endswith(".weight"):
|
||||||
|
key_lora = k[len("diffusion_model."):-len(".weight")]
|
||||||
|
key_map["base_model.model.{}".format(key_lora)] = k #official hunyuan lora format
|
||||||
|
|
||||||
return key_map
|
return key_map
|
||||||
|
|||||||
@ -5,11 +5,18 @@ from comfy.ldm.cascade.stage_c import StageC
|
|||||||
from comfy.ldm.cascade.stage_b import StageB
|
from comfy.ldm.cascade.stage_b import StageB
|
||||||
from comfy.ldm.modules.encoders.noise_aug_modules import CLIPEmbeddingNoiseAugmentation
|
from comfy.ldm.modules.encoders.noise_aug_modules import CLIPEmbeddingNoiseAugmentation
|
||||||
from comfy.ldm.modules.diffusionmodules.upscaling import ImageConcatWithNoiseAugmentation
|
from comfy.ldm.modules.diffusionmodules.upscaling import ImageConcatWithNoiseAugmentation
|
||||||
|
from comfy.ldm.modules.diffusionmodules.mmdit import OpenAISignatureMMDITWrapper
|
||||||
|
import comfy.ldm.aura.mmdit
|
||||||
|
import comfy.ldm.hydit.models
|
||||||
|
import comfy.ldm.audio.dit
|
||||||
|
import comfy.ldm.audio.embedders
|
||||||
import comfy.model_management
|
import comfy.model_management
|
||||||
import comfy.conds
|
import comfy.conds
|
||||||
import comfy.ops
|
import comfy.ops
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
from . import utils
|
from . import utils
|
||||||
|
import comfy.latent_formats
|
||||||
|
import math
|
||||||
|
|
||||||
class ModelType(Enum):
|
class ModelType(Enum):
|
||||||
EPS = 1
|
EPS = 1
|
||||||
@ -17,9 +24,11 @@ class ModelType(Enum):
|
|||||||
V_PREDICTION_EDM = 3
|
V_PREDICTION_EDM = 3
|
||||||
STABLE_CASCADE = 4
|
STABLE_CASCADE = 4
|
||||||
EDM = 5
|
EDM = 5
|
||||||
|
FLOW = 6
|
||||||
|
V_PREDICTION_CONTINUOUS = 7
|
||||||
|
|
||||||
|
|
||||||
from comfy.model_sampling import EPS, V_PREDICTION, EDM, ModelSamplingDiscrete, ModelSamplingContinuousEDM, StableCascadeSampling
|
from comfy.model_sampling import EPS, V_PREDICTION, EDM, ModelSamplingDiscrete, ModelSamplingContinuousEDM, StableCascadeSampling, ModelSamplingContinuousV
|
||||||
|
|
||||||
|
|
||||||
def model_sampling(model_config, model_type):
|
def model_sampling(model_config, model_type):
|
||||||
@ -32,12 +41,18 @@ def model_sampling(model_config, model_type):
|
|||||||
elif model_type == ModelType.V_PREDICTION_EDM:
|
elif model_type == ModelType.V_PREDICTION_EDM:
|
||||||
c = V_PREDICTION
|
c = V_PREDICTION
|
||||||
s = ModelSamplingContinuousEDM
|
s = ModelSamplingContinuousEDM
|
||||||
|
elif model_type == ModelType.FLOW:
|
||||||
|
c = comfy.model_sampling.CONST
|
||||||
|
s = comfy.model_sampling.ModelSamplingDiscreteFlow
|
||||||
elif model_type == ModelType.STABLE_CASCADE:
|
elif model_type == ModelType.STABLE_CASCADE:
|
||||||
c = EPS
|
c = EPS
|
||||||
s = StableCascadeSampling
|
s = StableCascadeSampling
|
||||||
elif model_type == ModelType.EDM:
|
elif model_type == ModelType.EDM:
|
||||||
c = EDM
|
c = EDM
|
||||||
s = ModelSamplingContinuousEDM
|
s = ModelSamplingContinuousEDM
|
||||||
|
elif model_type == ModelType.V_PREDICTION_CONTINUOUS:
|
||||||
|
c = V_PREDICTION
|
||||||
|
s = ModelSamplingContinuousV
|
||||||
|
|
||||||
class ModelSampling(s, c):
|
class ModelSampling(s, c):
|
||||||
pass
|
pass
|
||||||
@ -60,6 +75,9 @@ class BaseModel(torch.nn.Module):
|
|||||||
else:
|
else:
|
||||||
operations = comfy.ops.disable_weight_init
|
operations = comfy.ops.disable_weight_init
|
||||||
self.diffusion_model = unet_model(**unet_config, device=device, operations=operations)
|
self.diffusion_model = unet_model(**unet_config, device=device, operations=operations)
|
||||||
|
if comfy.model_management.force_channels_last():
|
||||||
|
self.diffusion_model.to(memory_format=torch.channels_last)
|
||||||
|
logging.debug("using channels last mode for diffusion model")
|
||||||
self.model_type = model_type
|
self.model_type = model_type
|
||||||
self.model_sampling = model_sampling(model_config, model_type)
|
self.model_sampling = model_sampling(model_config, model_type)
|
||||||
|
|
||||||
@ -201,9 +219,6 @@ class BaseModel(torch.nn.Module):
|
|||||||
unet_state_dict = self.diffusion_model.state_dict()
|
unet_state_dict = self.diffusion_model.state_dict()
|
||||||
unet_state_dict = self.model_config.process_unet_state_dict_for_saving(unet_state_dict)
|
unet_state_dict = self.model_config.process_unet_state_dict_for_saving(unet_state_dict)
|
||||||
|
|
||||||
if self.get_dtype() == torch.float16:
|
|
||||||
extra_sds = map(lambda sd: utils.convert_sd_to(sd, torch.float16), extra_sds)
|
|
||||||
|
|
||||||
if self.model_type == ModelType.V_PREDICTION:
|
if self.model_type == ModelType.V_PREDICTION:
|
||||||
unet_state_dict["v_pred"] = torch.tensor([])
|
unet_state_dict["v_pred"] = torch.tensor([])
|
||||||
|
|
||||||
@ -230,11 +245,11 @@ class BaseModel(torch.nn.Module):
|
|||||||
if self.manual_cast_dtype is not None:
|
if self.manual_cast_dtype is not None:
|
||||||
dtype = self.manual_cast_dtype
|
dtype = self.manual_cast_dtype
|
||||||
#TODO: this needs to be tweaked
|
#TODO: this needs to be tweaked
|
||||||
area = input_shape[0] * input_shape[2] * input_shape[3]
|
area = input_shape[0] * math.prod(input_shape[2:])
|
||||||
return (area * comfy.model_management.dtype_size(dtype) / 50) * (1024 * 1024)
|
return (area * comfy.model_management.dtype_size(dtype) / 50) * (1024 * 1024)
|
||||||
else:
|
else:
|
||||||
#TODO: this formula might be too aggressive since I tweaked the sub-quad and split algorithms to use less memory.
|
#TODO: this formula might be too aggressive since I tweaked the sub-quad and split algorithms to use less memory.
|
||||||
area = input_shape[0] * input_shape[2] * input_shape[3]
|
area = input_shape[0] * math.prod(input_shape[2:])
|
||||||
return (((area * 0.6) / 0.9) + 1024) * (1024 * 1024)
|
return (((area * 0.6) / 0.9) + 1024) * (1024 * 1024)
|
||||||
|
|
||||||
|
|
||||||
@ -557,3 +572,112 @@ class StableCascade_B(BaseModel):
|
|||||||
out["effnet"] = comfy.conds.CONDRegular(prior)
|
out["effnet"] = comfy.conds.CONDRegular(prior)
|
||||||
out["sca"] = comfy.conds.CONDRegular(torch.zeros((1,)))
|
out["sca"] = comfy.conds.CONDRegular(torch.zeros((1,)))
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class SD3(BaseModel):
|
||||||
|
def __init__(self, model_config, model_type=ModelType.FLOW, device=None):
|
||||||
|
super().__init__(model_config, model_type, device=device, unet_model=OpenAISignatureMMDITWrapper)
|
||||||
|
|
||||||
|
def encode_adm(self, **kwargs):
|
||||||
|
return kwargs["pooled_output"]
|
||||||
|
|
||||||
|
def extra_conds(self, **kwargs):
|
||||||
|
out = super().extra_conds(**kwargs)
|
||||||
|
cross_attn = kwargs.get("cross_attn", None)
|
||||||
|
if cross_attn is not None:
|
||||||
|
out['c_crossattn'] = comfy.conds.CONDRegular(cross_attn)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def memory_required(self, input_shape):
|
||||||
|
if comfy.model_management.xformers_enabled() or comfy.model_management.pytorch_attention_flash_attention():
|
||||||
|
dtype = self.get_dtype()
|
||||||
|
if self.manual_cast_dtype is not None:
|
||||||
|
dtype = self.manual_cast_dtype
|
||||||
|
#TODO: this probably needs to be tweaked
|
||||||
|
area = input_shape[0] * input_shape[2] * input_shape[3]
|
||||||
|
return (area * comfy.model_management.dtype_size(dtype) * 0.012) * (1024 * 1024)
|
||||||
|
else:
|
||||||
|
area = input_shape[0] * input_shape[2] * input_shape[3]
|
||||||
|
return (area * 0.3) * (1024 * 1024)
|
||||||
|
|
||||||
|
class AuraFlow(BaseModel):
|
||||||
|
def __init__(self, model_config, model_type=ModelType.FLOW, device=None):
|
||||||
|
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.aura.mmdit.MMDiT)
|
||||||
|
|
||||||
|
def extra_conds(self, **kwargs):
|
||||||
|
out = super().extra_conds(**kwargs)
|
||||||
|
cross_attn = kwargs.get("cross_attn", None)
|
||||||
|
if cross_attn is not None:
|
||||||
|
out['c_crossattn'] = comfy.conds.CONDRegular(cross_attn)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class StableAudio1(BaseModel):
|
||||||
|
def __init__(self, model_config, seconds_start_embedder_weights, seconds_total_embedder_weights, model_type=ModelType.V_PREDICTION_CONTINUOUS, device=None):
|
||||||
|
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.audio.dit.AudioDiffusionTransformer)
|
||||||
|
self.seconds_start_embedder = comfy.ldm.audio.embedders.NumberConditioner(768, min_val=0, max_val=512)
|
||||||
|
self.seconds_total_embedder = comfy.ldm.audio.embedders.NumberConditioner(768, min_val=0, max_val=512)
|
||||||
|
self.seconds_start_embedder.load_state_dict(seconds_start_embedder_weights)
|
||||||
|
self.seconds_total_embedder.load_state_dict(seconds_total_embedder_weights)
|
||||||
|
|
||||||
|
def extra_conds(self, **kwargs):
|
||||||
|
out = {}
|
||||||
|
|
||||||
|
noise = kwargs.get("noise", None)
|
||||||
|
device = kwargs["device"]
|
||||||
|
|
||||||
|
seconds_start = kwargs.get("seconds_start", 0)
|
||||||
|
seconds_total = kwargs.get("seconds_total", int(noise.shape[-1] / 21.53))
|
||||||
|
|
||||||
|
seconds_start_embed = self.seconds_start_embedder([seconds_start])[0].to(device)
|
||||||
|
seconds_total_embed = self.seconds_total_embedder([seconds_total])[0].to(device)
|
||||||
|
|
||||||
|
global_embed = torch.cat([seconds_start_embed, seconds_total_embed], dim=-1).reshape((1, -1))
|
||||||
|
out['global_embed'] = comfy.conds.CONDRegular(global_embed)
|
||||||
|
|
||||||
|
cross_attn = kwargs.get("cross_attn", None)
|
||||||
|
if cross_attn is not None:
|
||||||
|
cross_attn = torch.cat([cross_attn.to(device), seconds_start_embed.repeat((cross_attn.shape[0], 1, 1)), seconds_total_embed.repeat((cross_attn.shape[0], 1, 1))], dim=1)
|
||||||
|
out['c_crossattn'] = comfy.conds.CONDRegular(cross_attn)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def state_dict_for_saving(self, clip_state_dict=None, vae_state_dict=None, clip_vision_state_dict=None):
|
||||||
|
sd = super().state_dict_for_saving(clip_state_dict=clip_state_dict, vae_state_dict=vae_state_dict, clip_vision_state_dict=clip_vision_state_dict)
|
||||||
|
d = {"conditioner.conditioners.seconds_start.": self.seconds_start_embedder.state_dict(), "conditioner.conditioners.seconds_total.": self.seconds_total_embedder.state_dict()}
|
||||||
|
for k in d:
|
||||||
|
s = d[k]
|
||||||
|
for l in s:
|
||||||
|
sd["{}{}".format(k, l)] = s[l]
|
||||||
|
return sd
|
||||||
|
|
||||||
|
class HunyuanDiT(BaseModel):
|
||||||
|
def __init__(self, model_config, model_type=ModelType.V_PREDICTION, device=None):
|
||||||
|
super().__init__(model_config, model_type, device=device, unet_model=comfy.ldm.hydit.models.HunYuanDiT)
|
||||||
|
|
||||||
|
def extra_conds(self, **kwargs):
|
||||||
|
out = super().extra_conds(**kwargs)
|
||||||
|
cross_attn = kwargs.get("cross_attn", None)
|
||||||
|
if cross_attn is not None:
|
||||||
|
out['c_crossattn'] = comfy.conds.CONDRegular(cross_attn)
|
||||||
|
|
||||||
|
attention_mask = kwargs.get("attention_mask", None)
|
||||||
|
if attention_mask is not None:
|
||||||
|
out['text_embedding_mask'] = comfy.conds.CONDRegular(attention_mask)
|
||||||
|
|
||||||
|
conditioning_mt5xl = kwargs.get("conditioning_mt5xl", None)
|
||||||
|
if conditioning_mt5xl is not None:
|
||||||
|
out['encoder_hidden_states_t5'] = comfy.conds.CONDRegular(conditioning_mt5xl)
|
||||||
|
|
||||||
|
attention_mask_mt5xl = kwargs.get("attention_mask_mt5xl", None)
|
||||||
|
if attention_mask_mt5xl is not None:
|
||||||
|
out['text_embedding_mask_t5'] = comfy.conds.CONDRegular(attention_mask_mt5xl)
|
||||||
|
|
||||||
|
width = kwargs.get("width", 768)
|
||||||
|
height = kwargs.get("height", 768)
|
||||||
|
crop_w = kwargs.get("crop_w", 0)
|
||||||
|
crop_h = kwargs.get("crop_h", 0)
|
||||||
|
target_width = kwargs.get("target_width", width)
|
||||||
|
target_height = kwargs.get("target_height", height)
|
||||||
|
|
||||||
|
out['image_meta_size'] = comfy.conds.CONDRegular(torch.FloatTensor([[height, width, target_height, target_width, 0, 0]]))
|
||||||
|
return out
|
||||||
|
|||||||
@ -1,6 +1,9 @@
|
|||||||
import comfy.supported_models
|
import comfy.supported_models
|
||||||
import comfy.supported_models_base
|
import comfy.supported_models_base
|
||||||
|
import comfy.utils
|
||||||
|
import math
|
||||||
import logging
|
import logging
|
||||||
|
import torch
|
||||||
|
|
||||||
def count_blocks(state_dict_keys, prefix_string):
|
def count_blocks(state_dict_keys, prefix_string):
|
||||||
count = 0
|
count = 0
|
||||||
@ -26,12 +29,49 @@ def calculate_transformer_depth(prefix, state_dict_keys, state_dict):
|
|||||||
context_dim = state_dict['{}0.attn2.to_k.weight'.format(transformer_prefix)].shape[1]
|
context_dim = state_dict['{}0.attn2.to_k.weight'.format(transformer_prefix)].shape[1]
|
||||||
use_linear_in_transformer = len(state_dict['{}1.proj_in.weight'.format(prefix)].shape) == 2
|
use_linear_in_transformer = len(state_dict['{}1.proj_in.weight'.format(prefix)].shape) == 2
|
||||||
time_stack = '{}1.time_stack.0.attn1.to_q.weight'.format(prefix) in state_dict or '{}1.time_mix_blocks.0.attn1.to_q.weight'.format(prefix) in state_dict
|
time_stack = '{}1.time_stack.0.attn1.to_q.weight'.format(prefix) in state_dict or '{}1.time_mix_blocks.0.attn1.to_q.weight'.format(prefix) in state_dict
|
||||||
return last_transformer_depth, context_dim, use_linear_in_transformer, time_stack
|
time_stack_cross = '{}1.time_stack.0.attn2.to_q.weight'.format(prefix) in state_dict or '{}1.time_mix_blocks.0.attn2.to_q.weight'.format(prefix) in state_dict
|
||||||
|
return last_transformer_depth, context_dim, use_linear_in_transformer, time_stack, time_stack_cross
|
||||||
return None
|
return None
|
||||||
|
|
||||||
def detect_unet_config(state_dict, key_prefix):
|
def detect_unet_config(state_dict, key_prefix):
|
||||||
state_dict_keys = list(state_dict.keys())
|
state_dict_keys = list(state_dict.keys())
|
||||||
|
|
||||||
|
if '{}joint_blocks.0.context_block.attn.qkv.weight'.format(key_prefix) in state_dict_keys: #mmdit model
|
||||||
|
unet_config = {}
|
||||||
|
unet_config["in_channels"] = state_dict['{}x_embedder.proj.weight'.format(key_prefix)].shape[1]
|
||||||
|
patch_size = state_dict['{}x_embedder.proj.weight'.format(key_prefix)].shape[2]
|
||||||
|
unet_config["patch_size"] = patch_size
|
||||||
|
final_layer = '{}final_layer.linear.weight'.format(key_prefix)
|
||||||
|
if final_layer in state_dict:
|
||||||
|
unet_config["out_channels"] = state_dict[final_layer].shape[0] // (patch_size * patch_size)
|
||||||
|
|
||||||
|
unet_config["depth"] = state_dict['{}x_embedder.proj.weight'.format(key_prefix)].shape[0] // 64
|
||||||
|
unet_config["input_size"] = None
|
||||||
|
y_key = '{}y_embedder.mlp.0.weight'.format(key_prefix)
|
||||||
|
if y_key in state_dict_keys:
|
||||||
|
unet_config["adm_in_channels"] = state_dict[y_key].shape[1]
|
||||||
|
|
||||||
|
context_key = '{}context_embedder.weight'.format(key_prefix)
|
||||||
|
if context_key in state_dict_keys:
|
||||||
|
in_features = state_dict[context_key].shape[1]
|
||||||
|
out_features = state_dict[context_key].shape[0]
|
||||||
|
unet_config["context_embedder_config"] = {"target": "torch.nn.Linear", "params": {"in_features": in_features, "out_features": out_features}}
|
||||||
|
num_patches_key = '{}pos_embed'.format(key_prefix)
|
||||||
|
if num_patches_key in state_dict_keys:
|
||||||
|
num_patches = state_dict[num_patches_key].shape[1]
|
||||||
|
unet_config["num_patches"] = num_patches
|
||||||
|
unet_config["pos_embed_max_size"] = round(math.sqrt(num_patches))
|
||||||
|
|
||||||
|
rms_qk = '{}joint_blocks.0.context_block.attn.ln_q.weight'.format(key_prefix)
|
||||||
|
if rms_qk in state_dict_keys:
|
||||||
|
unet_config["qk_norm"] = "rms"
|
||||||
|
|
||||||
|
unet_config["pos_embed_scaling_factor"] = None #unused for inference
|
||||||
|
context_processor = '{}context_processor.layers.0.attn.qkv.weight'.format(key_prefix)
|
||||||
|
if context_processor in state_dict_keys:
|
||||||
|
unet_config["context_processor_layers"] = count_blocks(state_dict_keys, '{}context_processor.layers.'.format(key_prefix) + '{}.')
|
||||||
|
return unet_config
|
||||||
|
|
||||||
if '{}clf.1.weight'.format(key_prefix) in state_dict_keys: #stable cascade
|
if '{}clf.1.weight'.format(key_prefix) in state_dict_keys: #stable cascade
|
||||||
unet_config = {}
|
unet_config = {}
|
||||||
text_mapper_name = '{}clip_txt_mapper.weight'.format(key_prefix)
|
text_mapper_name = '{}clip_txt_mapper.weight'.format(key_prefix)
|
||||||
@ -58,9 +98,39 @@ def detect_unet_config(state_dict, key_prefix):
|
|||||||
unet_config['nhead'] = [-1, 9, 18, 18]
|
unet_config['nhead'] = [-1, 9, 18, 18]
|
||||||
unet_config['blocks'] = [[2, 4, 14, 4], [4, 14, 4, 2]]
|
unet_config['blocks'] = [[2, 4, 14, 4], [4, 14, 4, 2]]
|
||||||
unet_config['block_repeat'] = [[1, 1, 1, 1], [2, 2, 2, 2]]
|
unet_config['block_repeat'] = [[1, 1, 1, 1], [2, 2, 2, 2]]
|
||||||
|
|
||||||
return unet_config
|
return unet_config
|
||||||
|
|
||||||
|
if '{}transformer.rotary_pos_emb.inv_freq'.format(key_prefix) in state_dict_keys: #stable audio dit
|
||||||
|
unet_config = {}
|
||||||
|
unet_config["audio_model"] = "dit1.0"
|
||||||
|
return unet_config
|
||||||
|
|
||||||
|
if '{}double_layers.0.attn.w1q.weight'.format(key_prefix) in state_dict_keys: #aura flow dit
|
||||||
|
unet_config = {}
|
||||||
|
unet_config["max_seq"] = state_dict['{}positional_encoding'.format(key_prefix)].shape[1]
|
||||||
|
unet_config["cond_seq_dim"] = state_dict['{}cond_seq_linear.weight'.format(key_prefix)].shape[1]
|
||||||
|
double_layers = count_blocks(state_dict_keys, '{}double_layers.'.format(key_prefix) + '{}.')
|
||||||
|
single_layers = count_blocks(state_dict_keys, '{}single_layers.'.format(key_prefix) + '{}.')
|
||||||
|
unet_config["n_double_layers"] = double_layers
|
||||||
|
unet_config["n_layers"] = double_layers + single_layers
|
||||||
|
return unet_config
|
||||||
|
|
||||||
|
if '{}mlp_t5.0.weight'.format(key_prefix) in state_dict_keys: #Hunyuan DiT
|
||||||
|
unet_config = {}
|
||||||
|
unet_config["image_model"] = "hydit"
|
||||||
|
unet_config["depth"] = count_blocks(state_dict_keys, '{}blocks.'.format(key_prefix) + '{}.')
|
||||||
|
unet_config["hidden_size"] = state_dict['{}x_embedder.proj.weight'.format(key_prefix)].shape[0]
|
||||||
|
if unet_config["hidden_size"] == 1408 and unet_config["depth"] == 40: #DiT-g/2
|
||||||
|
unet_config["mlp_ratio"] = 4.3637
|
||||||
|
if state_dict['{}extra_embedder.0.weight'.format(key_prefix)].shape[1] == 3968:
|
||||||
|
unet_config["size_cond"] = True
|
||||||
|
unet_config["use_style_cond"] = True
|
||||||
|
unet_config["image_model"] = "hydit1"
|
||||||
|
return unet_config
|
||||||
|
|
||||||
|
if '{}input_blocks.0.0.weight'.format(key_prefix) not in state_dict_keys:
|
||||||
|
return None
|
||||||
|
|
||||||
unet_config = {
|
unet_config = {
|
||||||
"use_checkpoint": False,
|
"use_checkpoint": False,
|
||||||
"image_size": 32,
|
"image_size": 32,
|
||||||
@ -93,6 +163,7 @@ def detect_unet_config(state_dict, key_prefix):
|
|||||||
use_linear_in_transformer = False
|
use_linear_in_transformer = False
|
||||||
|
|
||||||
video_model = False
|
video_model = False
|
||||||
|
video_model_cross = False
|
||||||
|
|
||||||
current_res = 1
|
current_res = 1
|
||||||
count = 0
|
count = 0
|
||||||
@ -136,6 +207,7 @@ def detect_unet_config(state_dict, key_prefix):
|
|||||||
context_dim = out[1]
|
context_dim = out[1]
|
||||||
use_linear_in_transformer = out[2]
|
use_linear_in_transformer = out[2]
|
||||||
video_model = out[3]
|
video_model = out[3]
|
||||||
|
video_model_cross = out[4]
|
||||||
else:
|
else:
|
||||||
transformer_depth.append(0)
|
transformer_depth.append(0)
|
||||||
|
|
||||||
@ -176,6 +248,7 @@ def detect_unet_config(state_dict, key_prefix):
|
|||||||
unet_config["video_kernel_size"] = [3, 1, 1]
|
unet_config["video_kernel_size"] = [3, 1, 1]
|
||||||
unet_config["use_temporal_resblock"] = True
|
unet_config["use_temporal_resblock"] = True
|
||||||
unet_config["use_temporal_attention"] = True
|
unet_config["use_temporal_attention"] = True
|
||||||
|
unet_config["disable_temporal_crossattention"] = not video_model_cross
|
||||||
else:
|
else:
|
||||||
unet_config["use_temporal_resblock"] = False
|
unet_config["use_temporal_resblock"] = False
|
||||||
unet_config["use_temporal_attention"] = False
|
unet_config["use_temporal_attention"] = False
|
||||||
@ -192,12 +265,32 @@ def model_config_from_unet_config(unet_config, state_dict=None):
|
|||||||
|
|
||||||
def model_config_from_unet(state_dict, unet_key_prefix, use_base_if_no_match=False):
|
def model_config_from_unet(state_dict, unet_key_prefix, use_base_if_no_match=False):
|
||||||
unet_config = detect_unet_config(state_dict, unet_key_prefix)
|
unet_config = detect_unet_config(state_dict, unet_key_prefix)
|
||||||
|
if unet_config is None:
|
||||||
|
return None
|
||||||
model_config = model_config_from_unet_config(unet_config, state_dict)
|
model_config = model_config_from_unet_config(unet_config, state_dict)
|
||||||
if model_config is None and use_base_if_no_match:
|
if model_config is None and use_base_if_no_match:
|
||||||
return comfy.supported_models_base.BASE(unet_config)
|
return comfy.supported_models_base.BASE(unet_config)
|
||||||
else:
|
else:
|
||||||
return model_config
|
return model_config
|
||||||
|
|
||||||
|
def unet_prefix_from_state_dict(state_dict):
|
||||||
|
candidates = ["model.diffusion_model.", #ldm/sgm models
|
||||||
|
"model.model.", #audio models
|
||||||
|
]
|
||||||
|
counts = {k: 0 for k in candidates}
|
||||||
|
for k in state_dict:
|
||||||
|
for c in candidates:
|
||||||
|
if k.startswith(c):
|
||||||
|
counts[c] += 1
|
||||||
|
break
|
||||||
|
|
||||||
|
top = max(counts, key=counts.get)
|
||||||
|
if counts[top] > 5:
|
||||||
|
return top
|
||||||
|
else:
|
||||||
|
return "model." #aura flow and others
|
||||||
|
|
||||||
|
|
||||||
def convert_config(unet_config):
|
def convert_config(unet_config):
|
||||||
new_config = unet_config.copy()
|
new_config = unet_config.copy()
|
||||||
num_res_blocks = new_config.get("num_res_blocks", None)
|
num_res_blocks = new_config.get("num_res_blocks", None)
|
||||||
@ -381,3 +474,47 @@ def model_config_from_diffusers_unet(state_dict):
|
|||||||
if unet_config is not None:
|
if unet_config is not None:
|
||||||
return model_config_from_unet_config(unet_config)
|
return model_config_from_unet_config(unet_config)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
def convert_diffusers_mmdit(state_dict, output_prefix=""):
|
||||||
|
out_sd = {}
|
||||||
|
|
||||||
|
if 'transformer_blocks.0.attn.add_q_proj.weight' in state_dict: #SD3
|
||||||
|
num_blocks = count_blocks(state_dict, 'transformer_blocks.{}.')
|
||||||
|
depth = state_dict["pos_embed.proj.weight"].shape[0] // 64
|
||||||
|
sd_map = comfy.utils.mmdit_to_diffusers({"depth": depth, "num_blocks": num_blocks}, output_prefix=output_prefix)
|
||||||
|
elif 'joint_transformer_blocks.0.attn.add_k_proj.weight' in state_dict: #AuraFlow
|
||||||
|
num_joint = count_blocks(state_dict, 'joint_transformer_blocks.{}.')
|
||||||
|
num_single = count_blocks(state_dict, 'single_transformer_blocks.{}.')
|
||||||
|
sd_map = comfy.utils.auraflow_to_diffusers({"n_double_layers": num_joint, "n_layers": num_joint + num_single}, output_prefix=output_prefix)
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
|
||||||
|
for k in sd_map:
|
||||||
|
weight = state_dict.get(k, None)
|
||||||
|
if weight is not None:
|
||||||
|
t = sd_map[k]
|
||||||
|
|
||||||
|
if not isinstance(t, str):
|
||||||
|
if len(t) > 2:
|
||||||
|
fun = t[2]
|
||||||
|
else:
|
||||||
|
fun = lambda a: a
|
||||||
|
offset = t[1]
|
||||||
|
if offset is not None:
|
||||||
|
old_weight = out_sd.get(t[0], None)
|
||||||
|
if old_weight is None:
|
||||||
|
old_weight = torch.empty_like(weight)
|
||||||
|
old_weight = old_weight.repeat([3] + [1] * (len(old_weight.shape) - 1))
|
||||||
|
|
||||||
|
w = old_weight.narrow(offset[0], offset[1], offset[2])
|
||||||
|
else:
|
||||||
|
old_weight = weight
|
||||||
|
w = weight
|
||||||
|
w[:] = fun(weight)
|
||||||
|
t = t[0]
|
||||||
|
out_sd[t] = old_weight
|
||||||
|
else:
|
||||||
|
out_sd[t] = weight
|
||||||
|
state_dict.pop(k)
|
||||||
|
|
||||||
|
return out_sd
|
||||||
|
|||||||
@ -2,7 +2,6 @@ import psutil
|
|||||||
import logging
|
import logging
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
from comfy.cli_args import args
|
from comfy.cli_args import args
|
||||||
import comfy.utils
|
|
||||||
import torch
|
import torch
|
||||||
import sys
|
import sys
|
||||||
import platform
|
import platform
|
||||||
@ -168,7 +167,7 @@ if args.use_pytorch_cross_attention:
|
|||||||
ENABLE_PYTORCH_ATTENTION = True
|
ENABLE_PYTORCH_ATTENTION = True
|
||||||
XFORMERS_IS_AVAILABLE = False
|
XFORMERS_IS_AVAILABLE = False
|
||||||
|
|
||||||
VAE_DTYPE = torch.float32
|
VAE_DTYPES = [torch.float32]
|
||||||
|
|
||||||
try:
|
try:
|
||||||
if is_nvidia():
|
if is_nvidia():
|
||||||
@ -177,7 +176,7 @@ try:
|
|||||||
if ENABLE_PYTORCH_ATTENTION == False and args.use_split_cross_attention == False and args.use_quad_cross_attention == False:
|
if ENABLE_PYTORCH_ATTENTION == False and args.use_split_cross_attention == False and args.use_quad_cross_attention == False:
|
||||||
ENABLE_PYTORCH_ATTENTION = True
|
ENABLE_PYTORCH_ATTENTION = True
|
||||||
if torch.cuda.is_bf16_supported() and torch.cuda.get_device_properties(torch.cuda.current_device()).major >= 8:
|
if torch.cuda.is_bf16_supported() and torch.cuda.get_device_properties(torch.cuda.current_device()).major >= 8:
|
||||||
VAE_DTYPE = torch.bfloat16
|
VAE_DTYPES = [torch.bfloat16] + VAE_DTYPES
|
||||||
if is_intel_xpu():
|
if is_intel_xpu():
|
||||||
if args.use_split_cross_attention == False and args.use_quad_cross_attention == False:
|
if args.use_split_cross_attention == False and args.use_quad_cross_attention == False:
|
||||||
ENABLE_PYTORCH_ATTENTION = True
|
ENABLE_PYTORCH_ATTENTION = True
|
||||||
@ -185,17 +184,10 @@ except:
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
if is_intel_xpu():
|
if is_intel_xpu():
|
||||||
VAE_DTYPE = torch.bfloat16
|
VAE_DTYPES = [torch.bfloat16] + VAE_DTYPES
|
||||||
|
|
||||||
if args.cpu_vae:
|
if args.cpu_vae:
|
||||||
VAE_DTYPE = torch.float32
|
VAE_DTYPES = [torch.float32]
|
||||||
|
|
||||||
if args.fp16_vae:
|
|
||||||
VAE_DTYPE = torch.float16
|
|
||||||
elif args.bf16_vae:
|
|
||||||
VAE_DTYPE = torch.bfloat16
|
|
||||||
elif args.fp32_vae:
|
|
||||||
VAE_DTYPE = torch.float32
|
|
||||||
|
|
||||||
|
|
||||||
if ENABLE_PYTORCH_ATTENTION:
|
if ENABLE_PYTORCH_ATTENTION:
|
||||||
@ -259,7 +251,6 @@ try:
|
|||||||
except:
|
except:
|
||||||
logging.warning("Could not pick default device.")
|
logging.warning("Could not pick default device.")
|
||||||
|
|
||||||
logging.info("VAE dtype: {}".format(VAE_DTYPE))
|
|
||||||
|
|
||||||
current_loaded_models = []
|
current_loaded_models = []
|
||||||
|
|
||||||
@ -277,6 +268,7 @@ class LoadedModel:
|
|||||||
self.device = model.load_device
|
self.device = model.load_device
|
||||||
self.weights_loaded = False
|
self.weights_loaded = False
|
||||||
self.real_model = None
|
self.real_model = None
|
||||||
|
self.currently_used = True
|
||||||
|
|
||||||
def model_memory(self):
|
def model_memory(self):
|
||||||
return self.model.model_size()
|
return self.model.model_size()
|
||||||
@ -366,6 +358,7 @@ def free_memory(memory_required, device, keep_loaded=[]):
|
|||||||
if shift_model.device == device:
|
if shift_model.device == device:
|
||||||
if shift_model not in keep_loaded:
|
if shift_model not in keep_loaded:
|
||||||
can_unload.append((sys.getrefcount(shift_model.model), shift_model.model_memory(), i))
|
can_unload.append((sys.getrefcount(shift_model.model), shift_model.model_memory(), i))
|
||||||
|
shift_model.currently_used = False
|
||||||
|
|
||||||
for x in sorted(can_unload):
|
for x in sorted(can_unload):
|
||||||
i = x[-1]
|
i = x[-1]
|
||||||
@ -411,6 +404,7 @@ def load_models_gpu(models, memory_required=0, force_patch_weights=False):
|
|||||||
current_loaded_models.pop(loaded_model_index).model_unload(unpatch_weights=True)
|
current_loaded_models.pop(loaded_model_index).model_unload(unpatch_weights=True)
|
||||||
loaded = None
|
loaded = None
|
||||||
else:
|
else:
|
||||||
|
loaded.currently_used = True
|
||||||
models_already_loaded.append(loaded)
|
models_already_loaded.append(loaded)
|
||||||
|
|
||||||
if loaded is None:
|
if loaded is None:
|
||||||
@ -467,6 +461,16 @@ def load_models_gpu(models, memory_required=0, force_patch_weights=False):
|
|||||||
def load_model_gpu(model):
|
def load_model_gpu(model):
|
||||||
return load_models_gpu([model])
|
return load_models_gpu([model])
|
||||||
|
|
||||||
|
def loaded_models(only_currently_used=False):
|
||||||
|
output = []
|
||||||
|
for m in current_loaded_models:
|
||||||
|
if only_currently_used:
|
||||||
|
if not m.currently_used:
|
||||||
|
continue
|
||||||
|
|
||||||
|
output.append(m.model)
|
||||||
|
return output
|
||||||
|
|
||||||
def cleanup_models(keep_clone_weights_loaded=False):
|
def cleanup_models(keep_clone_weights_loaded=False):
|
||||||
to_delete = []
|
to_delete = []
|
||||||
for i in range(len(current_loaded_models)):
|
for i in range(len(current_loaded_models)):
|
||||||
@ -607,9 +611,22 @@ def vae_offload_device():
|
|||||||
else:
|
else:
|
||||||
return torch.device("cpu")
|
return torch.device("cpu")
|
||||||
|
|
||||||
def vae_dtype():
|
def vae_dtype(device=None, allowed_dtypes=[]):
|
||||||
global VAE_DTYPE
|
global VAE_DTYPES
|
||||||
return VAE_DTYPE
|
if args.fp16_vae:
|
||||||
|
return torch.float16
|
||||||
|
elif args.bf16_vae:
|
||||||
|
return torch.bfloat16
|
||||||
|
elif args.fp32_vae:
|
||||||
|
return torch.float32
|
||||||
|
|
||||||
|
for d in allowed_dtypes:
|
||||||
|
if d == torch.float16 and should_use_fp16(device, prioritize_performance=False):
|
||||||
|
return d
|
||||||
|
if d in VAE_DTYPES:
|
||||||
|
return d
|
||||||
|
|
||||||
|
return VAE_DTYPES[0]
|
||||||
|
|
||||||
def get_autocast_device(dev):
|
def get_autocast_device(dev):
|
||||||
if hasattr(dev, 'type'):
|
if hasattr(dev, 'type'):
|
||||||
@ -627,9 +644,32 @@ def supports_dtype(device, dtype): #TODO
|
|||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def supports_cast(device, dtype): #TODO
|
||||||
|
if dtype == torch.float32:
|
||||||
|
return True
|
||||||
|
if dtype == torch.float16:
|
||||||
|
return True
|
||||||
|
if is_device_mps(device):
|
||||||
|
return False
|
||||||
|
if directml_enabled: #TODO: test this
|
||||||
|
return False
|
||||||
|
if dtype == torch.bfloat16:
|
||||||
|
return True
|
||||||
|
if dtype == torch.float8_e4m3fn:
|
||||||
|
return True
|
||||||
|
if dtype == torch.float8_e5m2:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
def device_supports_non_blocking(device):
|
def device_supports_non_blocking(device):
|
||||||
if is_device_mps(device):
|
if is_device_mps(device):
|
||||||
return False #pytorch bug? mps doesn't support non blocking
|
return False #pytorch bug? mps doesn't support non blocking
|
||||||
|
if is_intel_xpu():
|
||||||
|
return False
|
||||||
|
if args.deterministic: #TODO: figure out why deterministic breaks non blocking from gpu to cpu (previews)
|
||||||
|
return False
|
||||||
|
if directml_enabled:
|
||||||
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
||||||
def device_should_use_non_blocking(device):
|
def device_should_use_non_blocking(device):
|
||||||
@ -638,6 +678,12 @@ def device_should_use_non_blocking(device):
|
|||||||
return False
|
return False
|
||||||
# return True #TODO: figure out why this causes memory issues on Nvidia and possibly others
|
# return True #TODO: figure out why this causes memory issues on Nvidia and possibly others
|
||||||
|
|
||||||
|
def force_channels_last():
|
||||||
|
if args.force_channels_last:
|
||||||
|
return True
|
||||||
|
|
||||||
|
#TODO
|
||||||
|
return False
|
||||||
|
|
||||||
def cast_to_device(tensor, device, dtype, copy=False):
|
def cast_to_device(tensor, device, dtype, copy=False):
|
||||||
device_supports_cast = False
|
device_supports_cast = False
|
||||||
@ -690,6 +736,8 @@ def pytorch_attention_flash_attention():
|
|||||||
#TODO: more reliable way of checking for flash attention?
|
#TODO: more reliable way of checking for flash attention?
|
||||||
if is_nvidia(): #pytorch flash attention only works on Nvidia
|
if is_nvidia(): #pytorch flash attention only works on Nvidia
|
||||||
return True
|
return True
|
||||||
|
if is_intel_xpu():
|
||||||
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
def force_upcast_attention_dtype():
|
def force_upcast_attention_dtype():
|
||||||
@ -876,6 +924,7 @@ def unload_all_models():
|
|||||||
|
|
||||||
|
|
||||||
def resolve_lowvram_weight(weight, model, key): #TODO: remove
|
def resolve_lowvram_weight(weight, model, key): #TODO: remove
|
||||||
|
print("WARNING: The comfy.model_management.resolve_lowvram_weight function will be removed soon, please stop using it.")
|
||||||
return weight
|
return weight
|
||||||
|
|
||||||
#TODO: might be cleaner to put this somewhere else
|
#TODO: might be cleaner to put this somewhere else
|
||||||
|
|||||||
@ -9,16 +9,26 @@ import comfy.model_management
|
|||||||
from comfy.types import UnetWrapperFunction
|
from comfy.types import UnetWrapperFunction
|
||||||
|
|
||||||
|
|
||||||
def apply_weight_decompose(dora_scale, weight):
|
def weight_decompose(dora_scale, weight, lora_diff, alpha, strength):
|
||||||
|
dora_scale = comfy.model_management.cast_to_device(dora_scale, weight.device, torch.float32)
|
||||||
|
lora_diff *= alpha
|
||||||
|
weight_calc = weight + lora_diff.type(weight.dtype)
|
||||||
weight_norm = (
|
weight_norm = (
|
||||||
weight.transpose(0, 1)
|
weight_calc.transpose(0, 1)
|
||||||
.reshape(weight.shape[1], -1)
|
.reshape(weight_calc.shape[1], -1)
|
||||||
.norm(dim=1, keepdim=True)
|
.norm(dim=1, keepdim=True)
|
||||||
.reshape(weight.shape[1], *[1] * (weight.dim() - 1))
|
.reshape(weight_calc.shape[1], *[1] * (weight_calc.dim() - 1))
|
||||||
.transpose(0, 1)
|
.transpose(0, 1)
|
||||||
)
|
)
|
||||||
|
|
||||||
return weight * (dora_scale / weight_norm).type(weight.dtype)
|
weight_calc *= (dora_scale / weight_norm).type(weight.dtype)
|
||||||
|
if strength != 1.0:
|
||||||
|
weight_calc -= weight
|
||||||
|
weight += strength * (weight_calc)
|
||||||
|
else:
|
||||||
|
weight[:] = weight_calc
|
||||||
|
return weight
|
||||||
|
|
||||||
|
|
||||||
def set_model_options_patch_replace(model_options, patch, name, block_name, number, transformer_index=None):
|
def set_model_options_patch_replace(model_options, patch, name, block_name, number, transformer_index=None):
|
||||||
to = model_options["transformer_options"].copy()
|
to = model_options["transformer_options"].copy()
|
||||||
@ -41,6 +51,18 @@ def set_model_options_patch_replace(model_options, patch, name, block_name, numb
|
|||||||
model_options["transformer_options"] = to
|
model_options["transformer_options"] = to
|
||||||
return model_options
|
return model_options
|
||||||
|
|
||||||
|
def set_model_options_post_cfg_function(model_options, post_cfg_function, disable_cfg1_optimization=False):
|
||||||
|
model_options["sampler_post_cfg_function"] = model_options.get("sampler_post_cfg_function", []) + [post_cfg_function]
|
||||||
|
if disable_cfg1_optimization:
|
||||||
|
model_options["disable_cfg1_optimization"] = True
|
||||||
|
return model_options
|
||||||
|
|
||||||
|
def set_model_options_pre_cfg_function(model_options, pre_cfg_function, disable_cfg1_optimization=False):
|
||||||
|
model_options["sampler_pre_cfg_function"] = model_options.get("sampler_pre_cfg_function", []) + [pre_cfg_function]
|
||||||
|
if disable_cfg1_optimization:
|
||||||
|
model_options["disable_cfg1_optimization"] = True
|
||||||
|
return model_options
|
||||||
|
|
||||||
class ModelPatcher:
|
class ModelPatcher:
|
||||||
def __init__(self, model, load_device, offload_device, size=0, current_device=None, weight_inplace_update=False):
|
def __init__(self, model, load_device, offload_device, size=0, current_device=None, weight_inplace_update=False):
|
||||||
self.size = size
|
self.size = size
|
||||||
@ -66,9 +88,7 @@ class ModelPatcher:
|
|||||||
def model_size(self):
|
def model_size(self):
|
||||||
if self.size > 0:
|
if self.size > 0:
|
||||||
return self.size
|
return self.size
|
||||||
model_sd = self.model.state_dict()
|
|
||||||
self.size = comfy.model_management.module_size(self.model)
|
self.size = comfy.model_management.module_size(self.model)
|
||||||
self.model_keys = set(model_sd.keys())
|
|
||||||
return self.size
|
return self.size
|
||||||
|
|
||||||
def clone(self):
|
def clone(self):
|
||||||
@ -80,7 +100,6 @@ class ModelPatcher:
|
|||||||
|
|
||||||
n.object_patches = self.object_patches.copy()
|
n.object_patches = self.object_patches.copy()
|
||||||
n.model_options = copy.deepcopy(self.model_options)
|
n.model_options = copy.deepcopy(self.model_options)
|
||||||
n.model_keys = self.model_keys
|
|
||||||
n.backup = self.backup
|
n.backup = self.backup
|
||||||
n.object_patches_backup = self.object_patches_backup
|
n.object_patches_backup = self.object_patches_backup
|
||||||
return n
|
return n
|
||||||
@ -115,9 +134,10 @@ class ModelPatcher:
|
|||||||
self.model_options["disable_cfg1_optimization"] = True
|
self.model_options["disable_cfg1_optimization"] = True
|
||||||
|
|
||||||
def set_model_sampler_post_cfg_function(self, post_cfg_function, disable_cfg1_optimization=False):
|
def set_model_sampler_post_cfg_function(self, post_cfg_function, disable_cfg1_optimization=False):
|
||||||
self.model_options["sampler_post_cfg_function"] = self.model_options.get("sampler_post_cfg_function", []) + [post_cfg_function]
|
self.model_options = set_model_options_post_cfg_function(self.model_options, post_cfg_function, disable_cfg1_optimization)
|
||||||
if disable_cfg1_optimization:
|
|
||||||
self.model_options["disable_cfg1_optimization"] = True
|
def set_model_sampler_pre_cfg_function(self, pre_cfg_function, disable_cfg1_optimization=False):
|
||||||
|
self.model_options = set_model_options_pre_cfg_function(self.model_options, pre_cfg_function, disable_cfg1_optimization)
|
||||||
|
|
||||||
def set_model_unet_function_wrapper(self, unet_wrapper_function: UnetWrapperFunction):
|
def set_model_unet_function_wrapper(self, unet_wrapper_function: UnetWrapperFunction):
|
||||||
self.model_options["model_function_wrapper"] = unet_wrapper_function
|
self.model_options["model_function_wrapper"] = unet_wrapper_function
|
||||||
@ -200,12 +220,23 @@ class ModelPatcher:
|
|||||||
|
|
||||||
def add_patches(self, patches, strength_patch=1.0, strength_model=1.0):
|
def add_patches(self, patches, strength_patch=1.0, strength_model=1.0):
|
||||||
p = set()
|
p = set()
|
||||||
|
model_sd = self.model.state_dict()
|
||||||
for k in patches:
|
for k in patches:
|
||||||
if k in self.model_keys:
|
offset = None
|
||||||
|
function = None
|
||||||
|
if isinstance(k, str):
|
||||||
|
key = k
|
||||||
|
else:
|
||||||
|
offset = k[1]
|
||||||
|
key = k[0]
|
||||||
|
if len(k) > 2:
|
||||||
|
function = k[2]
|
||||||
|
|
||||||
|
if key in model_sd:
|
||||||
p.add(k)
|
p.add(k)
|
||||||
current_patches = self.patches.get(k, [])
|
current_patches = self.patches.get(key, [])
|
||||||
current_patches.append((strength_patch, patches[k], strength_model))
|
current_patches.append((strength_patch, patches[k], strength_model, offset, function))
|
||||||
self.patches[k] = current_patches
|
self.patches[key] = current_patches
|
||||||
|
|
||||||
self.patches_uuid = uuid.uuid4()
|
self.patches_uuid = uuid.uuid4()
|
||||||
return list(p)
|
return list(p)
|
||||||
@ -320,7 +351,7 @@ class ModelPatcher:
|
|||||||
self.patch_weight_to_device(bias_key, device_to)
|
self.patch_weight_to_device(bias_key, device_to)
|
||||||
m.to(device_to)
|
m.to(device_to)
|
||||||
mem_counter += comfy.model_management.module_size(m)
|
mem_counter += comfy.model_management.module_size(m)
|
||||||
logging.debug("lowvram: loaded module regularly {}".format(m))
|
logging.debug("lowvram: loaded module regularly {} {}".format(n, m))
|
||||||
|
|
||||||
self.model_lowvram = True
|
self.model_lowvram = True
|
||||||
self.lowvram_patch_counter = patch_counter
|
self.lowvram_patch_counter = patch_counter
|
||||||
@ -328,9 +359,18 @@ class ModelPatcher:
|
|||||||
|
|
||||||
def calculate_weight(self, patches, weight, key):
|
def calculate_weight(self, patches, weight, key):
|
||||||
for p in patches:
|
for p in patches:
|
||||||
alpha = p[0]
|
strength = p[0]
|
||||||
v = p[1]
|
v = p[1]
|
||||||
strength_model = p[2]
|
strength_model = p[2]
|
||||||
|
offset = p[3]
|
||||||
|
function = p[4]
|
||||||
|
if function is None:
|
||||||
|
function = lambda a: a
|
||||||
|
|
||||||
|
old_weight = None
|
||||||
|
if offset is not None:
|
||||||
|
old_weight = weight
|
||||||
|
weight = weight.narrow(offset[0], offset[1], offset[2])
|
||||||
|
|
||||||
if strength_model != 1.0:
|
if strength_model != 1.0:
|
||||||
weight *= strength_model
|
weight *= strength_model
|
||||||
@ -346,26 +386,31 @@ class ModelPatcher:
|
|||||||
|
|
||||||
if patch_type == "diff":
|
if patch_type == "diff":
|
||||||
w1 = v[0]
|
w1 = v[0]
|
||||||
if alpha != 0.0:
|
if strength != 0.0:
|
||||||
if w1.shape != weight.shape:
|
if w1.shape != weight.shape:
|
||||||
logging.warning("WARNING SHAPE MISMATCH {} WEIGHT NOT MERGED {} != {}".format(key, w1.shape, weight.shape))
|
logging.warning("WARNING SHAPE MISMATCH {} WEIGHT NOT MERGED {} != {}".format(key, w1.shape, weight.shape))
|
||||||
else:
|
else:
|
||||||
weight += alpha * comfy.model_management.cast_to_device(w1, weight.device, weight.dtype)
|
weight += function(strength * comfy.model_management.cast_to_device(w1, weight.device, weight.dtype))
|
||||||
elif patch_type == "lora": #lora/locon
|
elif patch_type == "lora": #lora/locon
|
||||||
mat1 = comfy.model_management.cast_to_device(v[0], weight.device, torch.float32)
|
mat1 = comfy.model_management.cast_to_device(v[0], weight.device, torch.float32)
|
||||||
mat2 = comfy.model_management.cast_to_device(v[1], weight.device, torch.float32)
|
mat2 = comfy.model_management.cast_to_device(v[1], weight.device, torch.float32)
|
||||||
dora_scale = v[4]
|
dora_scale = v[4]
|
||||||
if v[2] is not None:
|
if v[2] is not None:
|
||||||
alpha *= v[2] / mat2.shape[0]
|
alpha = v[2] / mat2.shape[0]
|
||||||
|
else:
|
||||||
|
alpha = 1.0
|
||||||
|
|
||||||
if v[3] is not None:
|
if v[3] is not None:
|
||||||
#locon mid weights, hopefully the math is fine because I didn't properly test it
|
#locon mid weights, hopefully the math is fine because I didn't properly test it
|
||||||
mat3 = comfy.model_management.cast_to_device(v[3], weight.device, torch.float32)
|
mat3 = comfy.model_management.cast_to_device(v[3], weight.device, torch.float32)
|
||||||
final_shape = [mat2.shape[1], mat2.shape[0], mat3.shape[2], mat3.shape[3]]
|
final_shape = [mat2.shape[1], mat2.shape[0], mat3.shape[2], mat3.shape[3]]
|
||||||
mat2 = torch.mm(mat2.transpose(0, 1).flatten(start_dim=1), mat3.transpose(0, 1).flatten(start_dim=1)).reshape(final_shape).transpose(0, 1)
|
mat2 = torch.mm(mat2.transpose(0, 1).flatten(start_dim=1), mat3.transpose(0, 1).flatten(start_dim=1)).reshape(final_shape).transpose(0, 1)
|
||||||
try:
|
try:
|
||||||
weight += (alpha * torch.mm(mat1.flatten(start_dim=1), mat2.flatten(start_dim=1))).reshape(weight.shape).type(weight.dtype)
|
lora_diff = torch.mm(mat1.flatten(start_dim=1), mat2.flatten(start_dim=1)).reshape(weight.shape)
|
||||||
if dora_scale is not None:
|
if dora_scale is not None:
|
||||||
weight = apply_weight_decompose(comfy.model_management.cast_to_device(dora_scale, weight.device, torch.float32), weight)
|
weight = function(weight_decompose(dora_scale, weight, lora_diff, alpha, strength))
|
||||||
|
else:
|
||||||
|
weight += function(((strength * alpha) * lora_diff).type(weight.dtype))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
||||||
elif patch_type == "lokr":
|
elif patch_type == "lokr":
|
||||||
@ -402,19 +447,26 @@ class ModelPatcher:
|
|||||||
if len(w2.shape) == 4:
|
if len(w2.shape) == 4:
|
||||||
w1 = w1.unsqueeze(2).unsqueeze(2)
|
w1 = w1.unsqueeze(2).unsqueeze(2)
|
||||||
if v[2] is not None and dim is not None:
|
if v[2] is not None and dim is not None:
|
||||||
alpha *= v[2] / dim
|
alpha = v[2] / dim
|
||||||
|
else:
|
||||||
|
alpha = 1.0
|
||||||
|
|
||||||
try:
|
try:
|
||||||
weight += alpha * torch.kron(w1, w2).reshape(weight.shape).type(weight.dtype)
|
lora_diff = torch.kron(w1, w2).reshape(weight.shape)
|
||||||
if dora_scale is not None:
|
if dora_scale is not None:
|
||||||
weight = apply_weight_decompose(comfy.model_management.cast_to_device(dora_scale, weight.device, torch.float32), weight)
|
weight = function(weight_decompose(dora_scale, weight, lora_diff, alpha, strength))
|
||||||
|
else:
|
||||||
|
weight += function(((strength * alpha) * lora_diff).type(weight.dtype))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
||||||
elif patch_type == "loha":
|
elif patch_type == "loha":
|
||||||
w1a = v[0]
|
w1a = v[0]
|
||||||
w1b = v[1]
|
w1b = v[1]
|
||||||
if v[2] is not None:
|
if v[2] is not None:
|
||||||
alpha *= v[2] / w1b.shape[0]
|
alpha = v[2] / w1b.shape[0]
|
||||||
|
else:
|
||||||
|
alpha = 1.0
|
||||||
|
|
||||||
w2a = v[3]
|
w2a = v[3]
|
||||||
w2b = v[4]
|
w2b = v[4]
|
||||||
dora_scale = v[7]
|
dora_scale = v[7]
|
||||||
@ -437,14 +489,18 @@ class ModelPatcher:
|
|||||||
comfy.model_management.cast_to_device(w2b, weight.device, torch.float32))
|
comfy.model_management.cast_to_device(w2b, weight.device, torch.float32))
|
||||||
|
|
||||||
try:
|
try:
|
||||||
weight += (alpha * m1 * m2).reshape(weight.shape).type(weight.dtype)
|
lora_diff = (m1 * m2).reshape(weight.shape)
|
||||||
if dora_scale is not None:
|
if dora_scale is not None:
|
||||||
weight = apply_weight_decompose(comfy.model_management.cast_to_device(dora_scale, weight.device, torch.float32), weight)
|
weight = function(weight_decompose(dora_scale, weight, lora_diff, alpha, strength))
|
||||||
|
else:
|
||||||
|
weight += function(((strength * alpha) * lora_diff).type(weight.dtype))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
||||||
elif patch_type == "glora":
|
elif patch_type == "glora":
|
||||||
if v[4] is not None:
|
if v[4] is not None:
|
||||||
alpha *= v[4] / v[0].shape[0]
|
alpha = v[4] / v[0].shape[0]
|
||||||
|
else:
|
||||||
|
alpha = 1.0
|
||||||
|
|
||||||
dora_scale = v[5]
|
dora_scale = v[5]
|
||||||
|
|
||||||
@ -454,14 +510,19 @@ class ModelPatcher:
|
|||||||
b2 = comfy.model_management.cast_to_device(v[3].flatten(start_dim=1), weight.device, torch.float32)
|
b2 = comfy.model_management.cast_to_device(v[3].flatten(start_dim=1), weight.device, torch.float32)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
weight += ((torch.mm(b2, b1) + torch.mm(torch.mm(weight.flatten(start_dim=1), a2), a1)) * alpha).reshape(weight.shape).type(weight.dtype)
|
lora_diff = (torch.mm(b2, b1) + torch.mm(torch.mm(weight.flatten(start_dim=1), a2), a1)).reshape(weight.shape)
|
||||||
if dora_scale is not None:
|
if dora_scale is not None:
|
||||||
weight = apply_weight_decompose(comfy.model_management.cast_to_device(dora_scale, weight.device, torch.float32), weight)
|
weight = function(weight_decompose(dora_scale, weight, lora_diff, alpha, strength))
|
||||||
|
else:
|
||||||
|
weight += function(((strength * alpha) * lora_diff).type(weight.dtype))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
logging.error("ERROR {} {} {}".format(patch_type, key, e))
|
||||||
else:
|
else:
|
||||||
logging.warning("patch type not recognized {} {}".format(patch_type, key))
|
logging.warning("patch type not recognized {} {}".format(patch_type, key))
|
||||||
|
|
||||||
|
if old_weight is not None:
|
||||||
|
weight = old_weight
|
||||||
|
|
||||||
return weight
|
return weight
|
||||||
|
|
||||||
def unpatch_model(self, device_to=None, unpatch_weights=True):
|
def unpatch_model(self, device_to=None, unpatch_weights=True):
|
||||||
|
|||||||
@ -33,6 +33,19 @@ class EDM(V_PREDICTION):
|
|||||||
sigma = sigma.view(sigma.shape[:1] + (1,) * (model_output.ndim - 1))
|
sigma = sigma.view(sigma.shape[:1] + (1,) * (model_output.ndim - 1))
|
||||||
return model_input * self.sigma_data ** 2 / (sigma ** 2 + self.sigma_data ** 2) + model_output * sigma * self.sigma_data / (sigma ** 2 + self.sigma_data ** 2) ** 0.5
|
return model_input * self.sigma_data ** 2 / (sigma ** 2 + self.sigma_data ** 2) + model_output * sigma * self.sigma_data / (sigma ** 2 + self.sigma_data ** 2) ** 0.5
|
||||||
|
|
||||||
|
class CONST:
|
||||||
|
def calculate_input(self, sigma, noise):
|
||||||
|
return noise
|
||||||
|
|
||||||
|
def calculate_denoised(self, sigma, model_output, model_input):
|
||||||
|
sigma = sigma.view(sigma.shape[:1] + (1,) * (model_output.ndim - 1))
|
||||||
|
return model_input - model_output * sigma
|
||||||
|
|
||||||
|
def noise_scaling(self, sigma, noise, latent_image, max_denoise=False):
|
||||||
|
return sigma * noise + (1.0 - sigma) * latent_image
|
||||||
|
|
||||||
|
def inverse_noise_scaling(self, sigma, latent):
|
||||||
|
return latent / (1.0 - sigma)
|
||||||
|
|
||||||
class ModelSamplingDiscrete(torch.nn.Module):
|
class ModelSamplingDiscrete(torch.nn.Module):
|
||||||
def __init__(self, model_config=None):
|
def __init__(self, model_config=None):
|
||||||
@ -46,8 +59,9 @@ class ModelSamplingDiscrete(torch.nn.Module):
|
|||||||
beta_schedule = sampling_settings.get("beta_schedule", "linear")
|
beta_schedule = sampling_settings.get("beta_schedule", "linear")
|
||||||
linear_start = sampling_settings.get("linear_start", 0.00085)
|
linear_start = sampling_settings.get("linear_start", 0.00085)
|
||||||
linear_end = sampling_settings.get("linear_end", 0.012)
|
linear_end = sampling_settings.get("linear_end", 0.012)
|
||||||
|
timesteps = sampling_settings.get("timesteps", 1000)
|
||||||
|
|
||||||
self._register_schedule(given_betas=None, beta_schedule=beta_schedule, timesteps=1000, linear_start=linear_start, linear_end=linear_end, cosine_s=8e-3)
|
self._register_schedule(given_betas=None, beta_schedule=beta_schedule, timesteps=timesteps, linear_start=linear_start, linear_end=linear_end, cosine_s=8e-3)
|
||||||
self.sigma_data = 1.0
|
self.sigma_data = 1.0
|
||||||
|
|
||||||
def _register_schedule(self, given_betas=None, beta_schedule="linear", timesteps=1000,
|
def _register_schedule(self, given_betas=None, beta_schedule="linear", timesteps=1000,
|
||||||
@ -104,6 +118,12 @@ class ModelSamplingDiscrete(torch.nn.Module):
|
|||||||
percent = 1.0 - percent
|
percent = 1.0 - percent
|
||||||
return self.sigma(torch.tensor(percent * 999.0)).item()
|
return self.sigma(torch.tensor(percent * 999.0)).item()
|
||||||
|
|
||||||
|
class ModelSamplingDiscreteEDM(ModelSamplingDiscrete):
|
||||||
|
def timestep(self, sigma):
|
||||||
|
return 0.25 * sigma.log()
|
||||||
|
|
||||||
|
def sigma(self, timestep):
|
||||||
|
return (timestep / 0.25).exp()
|
||||||
|
|
||||||
class ModelSamplingContinuousEDM(torch.nn.Module):
|
class ModelSamplingContinuousEDM(torch.nn.Module):
|
||||||
def __init__(self, model_config=None):
|
def __init__(self, model_config=None):
|
||||||
@ -149,6 +169,57 @@ class ModelSamplingContinuousEDM(torch.nn.Module):
|
|||||||
log_sigma_min = math.log(self.sigma_min)
|
log_sigma_min = math.log(self.sigma_min)
|
||||||
return math.exp((math.log(self.sigma_max) - log_sigma_min) * percent + log_sigma_min)
|
return math.exp((math.log(self.sigma_max) - log_sigma_min) * percent + log_sigma_min)
|
||||||
|
|
||||||
|
|
||||||
|
class ModelSamplingContinuousV(ModelSamplingContinuousEDM):
|
||||||
|
def timestep(self, sigma):
|
||||||
|
return sigma.atan() / math.pi * 2
|
||||||
|
|
||||||
|
def sigma(self, timestep):
|
||||||
|
return (timestep * math.pi / 2).tan()
|
||||||
|
|
||||||
|
|
||||||
|
def time_snr_shift(alpha, t):
|
||||||
|
if alpha == 1.0:
|
||||||
|
return t
|
||||||
|
return alpha * t / (1 + (alpha - 1) * t)
|
||||||
|
|
||||||
|
class ModelSamplingDiscreteFlow(torch.nn.Module):
|
||||||
|
def __init__(self, model_config=None):
|
||||||
|
super().__init__()
|
||||||
|
if model_config is not None:
|
||||||
|
sampling_settings = model_config.sampling_settings
|
||||||
|
else:
|
||||||
|
sampling_settings = {}
|
||||||
|
|
||||||
|
self.set_parameters(shift=sampling_settings.get("shift", 1.0), multiplier=sampling_settings.get("multiplier", 1000))
|
||||||
|
|
||||||
|
def set_parameters(self, shift=1.0, timesteps=1000, multiplier=1000):
|
||||||
|
self.shift = shift
|
||||||
|
self.multiplier = multiplier
|
||||||
|
ts = self.sigma((torch.arange(1, timesteps + 1, 1) / timesteps) * multiplier)
|
||||||
|
self.register_buffer('sigmas', ts)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sigma_min(self):
|
||||||
|
return self.sigmas[0]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sigma_max(self):
|
||||||
|
return self.sigmas[-1]
|
||||||
|
|
||||||
|
def timestep(self, sigma):
|
||||||
|
return sigma * self.multiplier
|
||||||
|
|
||||||
|
def sigma(self, timestep):
|
||||||
|
return time_snr_shift(self.shift, timestep / self.multiplier)
|
||||||
|
|
||||||
|
def percent_to_sigma(self, percent):
|
||||||
|
if percent <= 0.0:
|
||||||
|
return 1.0
|
||||||
|
if percent >= 1.0:
|
||||||
|
return 0.0
|
||||||
|
return 1.0 - percent
|
||||||
|
|
||||||
class StableCascadeSampling(ModelSamplingDiscrete):
|
class StableCascadeSampling(ModelSamplingDiscrete):
|
||||||
def __init__(self, model_config=None):
|
def __init__(self, model_config=None):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
|
|||||||
41
comfy/ops.py
41
comfy/ops.py
@ -51,6 +51,20 @@ class disable_weight_init:
|
|||||||
else:
|
else:
|
||||||
return super().forward(*args, **kwargs)
|
return super().forward(*args, **kwargs)
|
||||||
|
|
||||||
|
class Conv1d(torch.nn.Conv1d, CastWeightBiasOp):
|
||||||
|
def reset_parameters(self):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def forward_comfy_cast_weights(self, input):
|
||||||
|
weight, bias = cast_bias_weight(self, input)
|
||||||
|
return self._conv_forward(input, weight, bias)
|
||||||
|
|
||||||
|
def forward(self, *args, **kwargs):
|
||||||
|
if self.comfy_cast_weights:
|
||||||
|
return self.forward_comfy_cast_weights(*args, **kwargs)
|
||||||
|
else:
|
||||||
|
return super().forward(*args, **kwargs)
|
||||||
|
|
||||||
class Conv2d(torch.nn.Conv2d, CastWeightBiasOp):
|
class Conv2d(torch.nn.Conv2d, CastWeightBiasOp):
|
||||||
def reset_parameters(self):
|
def reset_parameters(self):
|
||||||
return None
|
return None
|
||||||
@ -133,6 +147,27 @@ class disable_weight_init:
|
|||||||
else:
|
else:
|
||||||
return super().forward(*args, **kwargs)
|
return super().forward(*args, **kwargs)
|
||||||
|
|
||||||
|
class ConvTranspose1d(torch.nn.ConvTranspose1d, CastWeightBiasOp):
|
||||||
|
def reset_parameters(self):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def forward_comfy_cast_weights(self, input, output_size=None):
|
||||||
|
num_spatial_dims = 1
|
||||||
|
output_padding = self._output_padding(
|
||||||
|
input, output_size, self.stride, self.padding, self.kernel_size,
|
||||||
|
num_spatial_dims, self.dilation)
|
||||||
|
|
||||||
|
weight, bias = cast_bias_weight(self, input)
|
||||||
|
return torch.nn.functional.conv_transpose1d(
|
||||||
|
input, weight, bias, self.stride, self.padding,
|
||||||
|
output_padding, self.groups, self.dilation)
|
||||||
|
|
||||||
|
def forward(self, *args, **kwargs):
|
||||||
|
if self.comfy_cast_weights:
|
||||||
|
return self.forward_comfy_cast_weights(*args, **kwargs)
|
||||||
|
else:
|
||||||
|
return super().forward(*args, **kwargs)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def conv_nd(s, dims, *args, **kwargs):
|
def conv_nd(s, dims, *args, **kwargs):
|
||||||
if dims == 2:
|
if dims == 2:
|
||||||
@ -147,6 +182,9 @@ class manual_cast(disable_weight_init):
|
|||||||
class Linear(disable_weight_init.Linear):
|
class Linear(disable_weight_init.Linear):
|
||||||
comfy_cast_weights = True
|
comfy_cast_weights = True
|
||||||
|
|
||||||
|
class Conv1d(disable_weight_init.Conv1d):
|
||||||
|
comfy_cast_weights = True
|
||||||
|
|
||||||
class Conv2d(disable_weight_init.Conv2d):
|
class Conv2d(disable_weight_init.Conv2d):
|
||||||
comfy_cast_weights = True
|
comfy_cast_weights = True
|
||||||
|
|
||||||
@ -161,3 +199,6 @@ class manual_cast(disable_weight_init):
|
|||||||
|
|
||||||
class ConvTranspose2d(disable_weight_init.ConvTranspose2d):
|
class ConvTranspose2d(disable_weight_init.ConvTranspose2d):
|
||||||
comfy_cast_weights = True
|
comfy_cast_weights = True
|
||||||
|
|
||||||
|
class ConvTranspose1d(disable_weight_init.ConvTranspose1d):
|
||||||
|
comfy_cast_weights = True
|
||||||
|
|||||||
@ -24,6 +24,12 @@ def prepare_noise(latent_image, seed, noise_inds=None):
|
|||||||
noises = torch.cat(noises, axis=0)
|
noises = torch.cat(noises, axis=0)
|
||||||
return noises
|
return noises
|
||||||
|
|
||||||
|
def fix_empty_latent_channels(model, latent_image):
|
||||||
|
latent_channels = model.get_model_object("latent_format").latent_channels #Resize the empty latent image so it has the right number of channels
|
||||||
|
if latent_channels != latent_image.shape[1] and torch.count_nonzero(latent_image) == 0:
|
||||||
|
latent_image = comfy.utils.repeat_to_batch_size(latent_image, latent_channels, dim=1)
|
||||||
|
return latent_image
|
||||||
|
|
||||||
def prepare_sampling(model, noise_shape, positive, negative, noise_mask):
|
def prepare_sampling(model, noise_shape, positive, negative, noise_mask):
|
||||||
logging.warning("Warning: comfy.sample.prepare_sampling isn't used anymore and can be removed")
|
logging.warning("Warning: comfy.sample.prepare_sampling isn't used anymore and can be removed")
|
||||||
return model, positive, negative, noise_mask, []
|
return model, positive, negative, noise_mask, []
|
||||||
|
|||||||
@ -6,9 +6,12 @@ from comfy import model_management
|
|||||||
import math
|
import math
|
||||||
import logging
|
import logging
|
||||||
import comfy.sampler_helpers
|
import comfy.sampler_helpers
|
||||||
|
import scipy
|
||||||
|
import numpy
|
||||||
|
|
||||||
def get_area_and_mult(conds, x_in, timestep_in):
|
def get_area_and_mult(conds, x_in, timestep_in):
|
||||||
area = (x_in.shape[2], x_in.shape[3], 0, 0)
|
dims = tuple(x_in.shape[2:])
|
||||||
|
area = None
|
||||||
strength = 1.0
|
strength = 1.0
|
||||||
|
|
||||||
if 'timestep_start' in conds:
|
if 'timestep_start' in conds:
|
||||||
@ -20,11 +23,16 @@ def get_area_and_mult(conds, x_in, timestep_in):
|
|||||||
if timestep_in[0] < timestep_end:
|
if timestep_in[0] < timestep_end:
|
||||||
return None
|
return None
|
||||||
if 'area' in conds:
|
if 'area' in conds:
|
||||||
area = conds['area']
|
area = list(conds['area'])
|
||||||
if 'strength' in conds:
|
if 'strength' in conds:
|
||||||
strength = conds['strength']
|
strength = conds['strength']
|
||||||
|
|
||||||
input_x = x_in[:,:,area[2]:area[0] + area[2],area[3]:area[1] + area[3]]
|
input_x = x_in
|
||||||
|
if area is not None:
|
||||||
|
for i in range(len(dims)):
|
||||||
|
area[i] = min(input_x.shape[i + 2] - area[len(dims) + i], area[i])
|
||||||
|
input_x = input_x.narrow(i + 2, area[len(dims) + i], area[i])
|
||||||
|
|
||||||
if 'mask' in conds:
|
if 'mask' in conds:
|
||||||
# Scale the mask to the size of the input
|
# Scale the mask to the size of the input
|
||||||
# The mask should have been resized as we began the sampling process
|
# The mask should have been resized as we began the sampling process
|
||||||
@ -32,28 +40,30 @@ def get_area_and_mult(conds, x_in, timestep_in):
|
|||||||
if "mask_strength" in conds:
|
if "mask_strength" in conds:
|
||||||
mask_strength = conds["mask_strength"]
|
mask_strength = conds["mask_strength"]
|
||||||
mask = conds['mask']
|
mask = conds['mask']
|
||||||
assert(mask.shape[1] == x_in.shape[2])
|
assert(mask.shape[1:] == x_in.shape[2:])
|
||||||
assert(mask.shape[2] == x_in.shape[3])
|
|
||||||
mask = mask[:input_x.shape[0],area[2]:area[0] + area[2],area[3]:area[1] + area[3]] * mask_strength
|
mask = mask[:input_x.shape[0]]
|
||||||
|
if area is not None:
|
||||||
|
for i in range(len(dims)):
|
||||||
|
mask = mask.narrow(i + 1, area[len(dims) + i], area[i])
|
||||||
|
|
||||||
|
mask = mask * mask_strength
|
||||||
mask = mask.unsqueeze(1).repeat(input_x.shape[0] // mask.shape[0], input_x.shape[1], 1, 1)
|
mask = mask.unsqueeze(1).repeat(input_x.shape[0] // mask.shape[0], input_x.shape[1], 1, 1)
|
||||||
else:
|
else:
|
||||||
mask = torch.ones_like(input_x)
|
mask = torch.ones_like(input_x)
|
||||||
mult = mask * strength
|
mult = mask * strength
|
||||||
|
|
||||||
if 'mask' not in conds:
|
if 'mask' not in conds and area is not None:
|
||||||
rr = 8
|
rr = 8
|
||||||
if area[2] != 0:
|
for i in range(len(dims)):
|
||||||
for t in range(rr):
|
if area[len(dims) + i] != 0:
|
||||||
mult[:,:,t:1+t,:] *= ((1.0/rr) * (t + 1))
|
for t in range(rr):
|
||||||
if (area[0] + area[2]) < x_in.shape[2]:
|
m = mult.narrow(i + 2, t, 1)
|
||||||
for t in range(rr):
|
m *= ((1.0/rr) * (t + 1))
|
||||||
mult[:,:,area[0] - 1 - t:area[0] - t,:] *= ((1.0/rr) * (t + 1))
|
if (area[i] + area[len(dims) + i]) < x_in.shape[i + 2]:
|
||||||
if area[3] != 0:
|
for t in range(rr):
|
||||||
for t in range(rr):
|
m = mult.narrow(i + 2, area[i] - 1 - t, 1)
|
||||||
mult[:,:,:,t:1+t] *= ((1.0/rr) * (t + 1))
|
m *= ((1.0/rr) * (t + 1))
|
||||||
if (area[1] + area[3]) < x_in.shape[3]:
|
|
||||||
for t in range(rr):
|
|
||||||
mult[:,:,:,area[1] - 1 - t:area[1] - t] *= ((1.0/rr) * (t + 1))
|
|
||||||
|
|
||||||
conditioning = {}
|
conditioning = {}
|
||||||
model_conds = conds["model_conds"]
|
model_conds = conds["model_conds"]
|
||||||
@ -219,8 +229,19 @@ def calc_cond_batch(model, conds, x_in, timestep, model_options):
|
|||||||
|
|
||||||
for o in range(batch_chunks):
|
for o in range(batch_chunks):
|
||||||
cond_index = cond_or_uncond[o]
|
cond_index = cond_or_uncond[o]
|
||||||
out_conds[cond_index][:,:,area[o][2]:area[o][0] + area[o][2],area[o][3]:area[o][1] + area[o][3]] += output[o] * mult[o]
|
a = area[o]
|
||||||
out_counts[cond_index][:,:,area[o][2]:area[o][0] + area[o][2],area[o][3]:area[o][1] + area[o][3]] += mult[o]
|
if a is None:
|
||||||
|
out_conds[cond_index] += output[o] * mult[o]
|
||||||
|
out_counts[cond_index] += mult[o]
|
||||||
|
else:
|
||||||
|
out_c = out_conds[cond_index]
|
||||||
|
out_cts = out_counts[cond_index]
|
||||||
|
dims = len(a) // 2
|
||||||
|
for i in range(dims):
|
||||||
|
out_c = out_c.narrow(i + 2, a[i + dims], a[i])
|
||||||
|
out_cts = out_cts.narrow(i + 2, a[i + dims], a[i])
|
||||||
|
out_c += output[o] * mult[o]
|
||||||
|
out_cts += mult[o]
|
||||||
|
|
||||||
for i in range(len(out_conds)):
|
for i in range(len(out_conds)):
|
||||||
out_conds[i] /= out_counts[i]
|
out_conds[i] /= out_counts[i]
|
||||||
@ -256,6 +277,12 @@ def sampling_function(model, x, timestep, uncond, cond, cond_scale, model_option
|
|||||||
|
|
||||||
conds = [cond, uncond_]
|
conds = [cond, uncond_]
|
||||||
out = calc_cond_batch(model, conds, x, timestep, model_options)
|
out = calc_cond_batch(model, conds, x, timestep, model_options)
|
||||||
|
|
||||||
|
for fn in model_options.get("sampler_pre_cfg_function", []):
|
||||||
|
args = {"conds":conds, "conds_out": out, "cond_scale": cond_scale, "timestep": timestep,
|
||||||
|
"input": x, "sigma": timestep, "model": model, "model_options": model_options}
|
||||||
|
out = fn(args)
|
||||||
|
|
||||||
return cfg_function(model, out[0], out[1], cond_scale, x, timestep, model_options=model_options, cond=cond, uncond=uncond_)
|
return cfg_function(model, out[0], out[1], cond_scale, x, timestep, model_options=model_options, cond=cond, uncond=uncond_)
|
||||||
|
|
||||||
|
|
||||||
@ -286,13 +313,18 @@ def simple_scheduler(model_sampling, steps):
|
|||||||
def ddim_scheduler(model_sampling, steps):
|
def ddim_scheduler(model_sampling, steps):
|
||||||
s = model_sampling
|
s = model_sampling
|
||||||
sigs = []
|
sigs = []
|
||||||
ss = max(len(s.sigmas) // steps, 1)
|
|
||||||
x = 1
|
x = 1
|
||||||
|
if math.isclose(float(s.sigmas[x]), 0, abs_tol=0.00001):
|
||||||
|
steps += 1
|
||||||
|
sigs = []
|
||||||
|
else:
|
||||||
|
sigs = [0.0]
|
||||||
|
|
||||||
|
ss = max(len(s.sigmas) // steps, 1)
|
||||||
while x < len(s.sigmas):
|
while x < len(s.sigmas):
|
||||||
sigs += [float(s.sigmas[x])]
|
sigs += [float(s.sigmas[x])]
|
||||||
x += ss
|
x += ss
|
||||||
sigs = sigs[::-1]
|
sigs = sigs[::-1]
|
||||||
sigs += [0.0]
|
|
||||||
return torch.FloatTensor(sigs)
|
return torch.FloatTensor(sigs)
|
||||||
|
|
||||||
def normal_scheduler(model_sampling, steps, sgm=False, floor=False):
|
def normal_scheduler(model_sampling, steps, sgm=False, floor=False):
|
||||||
@ -300,15 +332,34 @@ def normal_scheduler(model_sampling, steps, sgm=False, floor=False):
|
|||||||
start = s.timestep(s.sigma_max)
|
start = s.timestep(s.sigma_max)
|
||||||
end = s.timestep(s.sigma_min)
|
end = s.timestep(s.sigma_min)
|
||||||
|
|
||||||
|
append_zero = True
|
||||||
if sgm:
|
if sgm:
|
||||||
timesteps = torch.linspace(start, end, steps + 1)[:-1]
|
timesteps = torch.linspace(start, end, steps + 1)[:-1]
|
||||||
else:
|
else:
|
||||||
|
if math.isclose(float(s.sigma(end)), 0, abs_tol=0.00001):
|
||||||
|
steps += 1
|
||||||
|
append_zero = False
|
||||||
timesteps = torch.linspace(start, end, steps)
|
timesteps = torch.linspace(start, end, steps)
|
||||||
|
|
||||||
sigs = []
|
sigs = []
|
||||||
for x in range(len(timesteps)):
|
for x in range(len(timesteps)):
|
||||||
ts = timesteps[x]
|
ts = timesteps[x]
|
||||||
sigs.append(s.sigma(ts))
|
sigs.append(float(s.sigma(ts)))
|
||||||
|
|
||||||
|
if append_zero:
|
||||||
|
sigs += [0.0]
|
||||||
|
|
||||||
|
return torch.FloatTensor(sigs)
|
||||||
|
|
||||||
|
# Implemented based on: https://arxiv.org/abs/2407.12173
|
||||||
|
def beta_scheduler(model_sampling, steps, alpha=0.6, beta=0.6):
|
||||||
|
total_timesteps = (len(model_sampling.sigmas) - 1)
|
||||||
|
ts = 1 - numpy.linspace(0, 1, steps, endpoint=False)
|
||||||
|
ts = numpy.rint(scipy.stats.beta.ppf(ts, alpha, beta) * total_timesteps)
|
||||||
|
|
||||||
|
sigs = []
|
||||||
|
for t in ts:
|
||||||
|
sigs += [float(model_sampling.sigmas[int(t)])]
|
||||||
sigs += [0.0]
|
sigs += [0.0]
|
||||||
return torch.FloatTensor(sigs)
|
return torch.FloatTensor(sigs)
|
||||||
|
|
||||||
@ -335,7 +386,7 @@ def get_mask_aabb(masks):
|
|||||||
|
|
||||||
return bounding_boxes, is_empty
|
return bounding_boxes, is_empty
|
||||||
|
|
||||||
def resolve_areas_and_cond_masks(conditions, h, w, device):
|
def resolve_areas_and_cond_masks_multidim(conditions, dims, device):
|
||||||
# We need to decide on an area outside the sampling loop in order to properly generate opposite areas of equal sizes.
|
# We need to decide on an area outside the sampling loop in order to properly generate opposite areas of equal sizes.
|
||||||
# While we're doing this, we can also resolve the mask device and scaling for performance reasons
|
# While we're doing this, we can also resolve the mask device and scaling for performance reasons
|
||||||
for i in range(len(conditions)):
|
for i in range(len(conditions)):
|
||||||
@ -344,7 +395,14 @@ def resolve_areas_and_cond_masks(conditions, h, w, device):
|
|||||||
area = c['area']
|
area = c['area']
|
||||||
if area[0] == "percentage":
|
if area[0] == "percentage":
|
||||||
modified = c.copy()
|
modified = c.copy()
|
||||||
area = (max(1, round(area[1] * h)), max(1, round(area[2] * w)), round(area[3] * h), round(area[4] * w))
|
a = area[1:]
|
||||||
|
a_len = len(a) // 2
|
||||||
|
area = ()
|
||||||
|
for d in range(len(dims)):
|
||||||
|
area += (max(1, round(a[d] * dims[d])),)
|
||||||
|
for d in range(len(dims)):
|
||||||
|
area += (round(a[d + a_len] * dims[d]),)
|
||||||
|
|
||||||
modified['area'] = area
|
modified['area'] = area
|
||||||
c = modified
|
c = modified
|
||||||
conditions[i] = c
|
conditions[i] = c
|
||||||
@ -353,12 +411,12 @@ def resolve_areas_and_cond_masks(conditions, h, w, device):
|
|||||||
mask = c['mask']
|
mask = c['mask']
|
||||||
mask = mask.to(device=device)
|
mask = mask.to(device=device)
|
||||||
modified = c.copy()
|
modified = c.copy()
|
||||||
if len(mask.shape) == 2:
|
if len(mask.shape) == len(dims):
|
||||||
mask = mask.unsqueeze(0)
|
mask = mask.unsqueeze(0)
|
||||||
if mask.shape[1] != h or mask.shape[2] != w:
|
if mask.shape[1:] != dims:
|
||||||
mask = torch.nn.functional.interpolate(mask.unsqueeze(1), size=(h, w), mode='bilinear', align_corners=False).squeeze(1)
|
mask = torch.nn.functional.interpolate(mask.unsqueeze(1), size=dims, mode='bilinear', align_corners=False).squeeze(1)
|
||||||
|
|
||||||
if modified.get("set_area_to_bounds", False):
|
if modified.get("set_area_to_bounds", False): #TODO: handle dim != 2
|
||||||
bounds = torch.max(torch.abs(mask),dim=0).values.unsqueeze(0)
|
bounds = torch.max(torch.abs(mask),dim=0).values.unsqueeze(0)
|
||||||
boxes, is_empty = get_mask_aabb(bounds)
|
boxes, is_empty = get_mask_aabb(bounds)
|
||||||
if is_empty[0]:
|
if is_empty[0]:
|
||||||
@ -375,7 +433,11 @@ def resolve_areas_and_cond_masks(conditions, h, w, device):
|
|||||||
modified['mask'] = mask
|
modified['mask'] = mask
|
||||||
conditions[i] = modified
|
conditions[i] = modified
|
||||||
|
|
||||||
def create_cond_with_same_area_if_none(conds, c):
|
def resolve_areas_and_cond_masks(conditions, h, w, device):
|
||||||
|
logging.warning("WARNING: The comfy.samplers.resolve_areas_and_cond_masks function is deprecated please use the resolve_areas_and_cond_masks_multidim one instead.")
|
||||||
|
return resolve_areas_and_cond_masks_multidim(conditions, [h, w], device)
|
||||||
|
|
||||||
|
def create_cond_with_same_area_if_none(conds, c): #TODO: handle dim != 2
|
||||||
if 'area' not in c:
|
if 'area' not in c:
|
||||||
return
|
return
|
||||||
|
|
||||||
@ -479,7 +541,10 @@ def encode_model_conds(model_function, conds, noise, device, prompt_type, **kwar
|
|||||||
params = x.copy()
|
params = x.copy()
|
||||||
params["device"] = device
|
params["device"] = device
|
||||||
params["noise"] = noise
|
params["noise"] = noise
|
||||||
params["width"] = params.get("width", noise.shape[3] * 8)
|
default_width = None
|
||||||
|
if len(noise.shape) >= 4: #TODO: 8 multiple should be set by the model
|
||||||
|
default_width = noise.shape[3] * 8
|
||||||
|
params["width"] = params.get("width", default_width)
|
||||||
params["height"] = params.get("height", noise.shape[2] * 8)
|
params["height"] = params.get("height", noise.shape[2] * 8)
|
||||||
params["prompt_type"] = params.get("prompt_type", prompt_type)
|
params["prompt_type"] = params.get("prompt_type", prompt_type)
|
||||||
for k in kwargs:
|
for k in kwargs:
|
||||||
@ -504,9 +569,10 @@ class Sampler:
|
|||||||
sigma = float(sigmas[0])
|
sigma = float(sigmas[0])
|
||||||
return math.isclose(max_sigma, sigma, rel_tol=1e-05) or sigma > max_sigma
|
return math.isclose(max_sigma, sigma, rel_tol=1e-05) or sigma > max_sigma
|
||||||
|
|
||||||
KSAMPLER_NAMES = ["euler", "euler_ancestral", "heun", "heunpp2","dpm_2", "dpm_2_ancestral",
|
KSAMPLER_NAMES = ["euler", "euler_cfg_pp", "euler_ancestral", "euler_ancestral_cfg_pp", "heun", "heunpp2","dpm_2", "dpm_2_ancestral",
|
||||||
"lms", "dpm_fast", "dpm_adaptive", "dpmpp_2s_ancestral", "dpmpp_sde", "dpmpp_sde_gpu",
|
"lms", "dpm_fast", "dpm_adaptive", "dpmpp_2s_ancestral", "dpmpp_sde", "dpmpp_sde_gpu",
|
||||||
"dpmpp_2m", "dpmpp_2m_sde", "dpmpp_2m_sde_gpu", "dpmpp_3m_sde", "dpmpp_3m_sde_gpu", "ddpm", "lcm"]
|
"dpmpp_2m", "dpmpp_2m_sde", "dpmpp_2m_sde_gpu", "dpmpp_3m_sde", "dpmpp_3m_sde_gpu", "ddpm", "lcm",
|
||||||
|
"ipndm", "ipndm_v", "deis"]
|
||||||
|
|
||||||
class KSAMPLER(Sampler):
|
class KSAMPLER(Sampler):
|
||||||
def __init__(self, sampler_function, extra_options={}, inpaint_options={}):
|
def __init__(self, sampler_function, extra_options={}, inpaint_options={}):
|
||||||
@ -567,7 +633,7 @@ def ksampler(sampler_name, extra_options={}, inpaint_options={}):
|
|||||||
def process_conds(model, noise, conds, device, latent_image=None, denoise_mask=None, seed=None):
|
def process_conds(model, noise, conds, device, latent_image=None, denoise_mask=None, seed=None):
|
||||||
for k in conds:
|
for k in conds:
|
||||||
conds[k] = conds[k][:]
|
conds[k] = conds[k][:]
|
||||||
resolve_areas_and_cond_masks(conds[k], noise.shape[2], noise.shape[3], device)
|
resolve_areas_and_cond_masks_multidim(conds[k], noise.shape[2:], device)
|
||||||
|
|
||||||
for k in conds:
|
for k in conds:
|
||||||
calculate_start_end_timesteps(model, conds[k])
|
calculate_start_end_timesteps(model, conds[k])
|
||||||
@ -663,7 +729,7 @@ def sample(model, noise, positive, negative, cfg, device, sampler, sigmas, model
|
|||||||
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
|
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
|
||||||
|
|
||||||
|
|
||||||
SCHEDULER_NAMES = ["normal", "karras", "exponential", "sgm_uniform", "simple", "ddim_uniform"]
|
SCHEDULER_NAMES = ["normal", "karras", "exponential", "sgm_uniform", "simple", "ddim_uniform", "beta"]
|
||||||
SAMPLER_NAMES = KSAMPLER_NAMES + ["ddim", "uni_pc", "uni_pc_bh2"]
|
SAMPLER_NAMES = KSAMPLER_NAMES + ["ddim", "uni_pc", "uni_pc_bh2"]
|
||||||
|
|
||||||
def calculate_sigmas(model_sampling, scheduler_name, steps):
|
def calculate_sigmas(model_sampling, scheduler_name, steps):
|
||||||
@ -679,6 +745,8 @@ def calculate_sigmas(model_sampling, scheduler_name, steps):
|
|||||||
sigmas = ddim_scheduler(model_sampling, steps)
|
sigmas = ddim_scheduler(model_sampling, steps)
|
||||||
elif scheduler_name == "sgm_uniform":
|
elif scheduler_name == "sgm_uniform":
|
||||||
sigmas = normal_scheduler(model_sampling, steps, sgm=True)
|
sigmas = normal_scheduler(model_sampling, steps, sgm=True)
|
||||||
|
elif scheduler_name == "beta":
|
||||||
|
sigmas = beta_scheduler(model_sampling, steps)
|
||||||
else:
|
else:
|
||||||
logging.error("error invalid scheduler {}".format(scheduler_name))
|
logging.error("error invalid scheduler {}".format(scheduler_name))
|
||||||
return sigmas
|
return sigmas
|
||||||
|
|||||||
219
comfy/sd.py
219
comfy/sd.py
@ -6,7 +6,7 @@ from comfy import model_management
|
|||||||
from .ldm.models.autoencoder import AutoencoderKL, AutoencodingEngine
|
from .ldm.models.autoencoder import AutoencoderKL, AutoencodingEngine
|
||||||
from .ldm.cascade.stage_a import StageA
|
from .ldm.cascade.stage_a import StageA
|
||||||
from .ldm.cascade.stage_c_coder import StageC_coder
|
from .ldm.cascade.stage_c_coder import StageC_coder
|
||||||
|
from .ldm.audio.autoencoder import AudioOobleckVAE
|
||||||
import yaml
|
import yaml
|
||||||
|
|
||||||
import comfy.utils
|
import comfy.utils
|
||||||
@ -14,12 +14,15 @@ import comfy.utils
|
|||||||
from . import clip_vision
|
from . import clip_vision
|
||||||
from . import gligen
|
from . import gligen
|
||||||
from . import diffusers_convert
|
from . import diffusers_convert
|
||||||
from . import model_base
|
|
||||||
from . import model_detection
|
from . import model_detection
|
||||||
|
|
||||||
from . import sd1_clip
|
from . import sd1_clip
|
||||||
from . import sd2_clip
|
from . import sd2_clip
|
||||||
from . import sdxl_clip
|
from . import sdxl_clip
|
||||||
|
import comfy.text_encoders.sd3_clip
|
||||||
|
import comfy.text_encoders.sa_t5
|
||||||
|
import comfy.text_encoders.aura_t5
|
||||||
|
import comfy.text_encoders.hydit
|
||||||
|
|
||||||
import comfy.model_patcher
|
import comfy.model_patcher
|
||||||
import comfy.lora
|
import comfy.lora
|
||||||
@ -27,36 +30,6 @@ import comfy.t2i_adapter.adapter
|
|||||||
import comfy.supported_models_base
|
import comfy.supported_models_base
|
||||||
import comfy.taesd.taesd
|
import comfy.taesd.taesd
|
||||||
|
|
||||||
def load_model_weights(model, sd):
|
|
||||||
m, u = model.load_state_dict(sd, strict=False)
|
|
||||||
m = set(m)
|
|
||||||
unexpected_keys = set(u)
|
|
||||||
|
|
||||||
k = list(sd.keys())
|
|
||||||
for x in k:
|
|
||||||
if x not in unexpected_keys:
|
|
||||||
w = sd.pop(x)
|
|
||||||
del w
|
|
||||||
if len(m) > 0:
|
|
||||||
logging.warning("missing {}".format(m))
|
|
||||||
return model
|
|
||||||
|
|
||||||
def load_clip_weights(model, sd):
|
|
||||||
k = list(sd.keys())
|
|
||||||
for x in k:
|
|
||||||
if x.startswith("cond_stage_model.transformer.") and not x.startswith("cond_stage_model.transformer.text_model."):
|
|
||||||
y = x.replace("cond_stage_model.transformer.", "cond_stage_model.transformer.text_model.")
|
|
||||||
sd[y] = sd.pop(x)
|
|
||||||
|
|
||||||
if 'cond_stage_model.transformer.text_model.embeddings.position_ids' in sd:
|
|
||||||
ids = sd['cond_stage_model.transformer.text_model.embeddings.position_ids']
|
|
||||||
if ids.dtype == torch.float32:
|
|
||||||
sd['cond_stage_model.transformer.text_model.embeddings.position_ids'] = ids.round()
|
|
||||||
|
|
||||||
sd = comfy.utils.clip_text_transformers_convert(sd, "cond_stage_model.model.", "cond_stage_model.transformer.")
|
|
||||||
return load_model_weights(model, sd)
|
|
||||||
|
|
||||||
|
|
||||||
def load_lora_for_models(model, clip, lora, strength_model, strength_clip):
|
def load_lora_for_models(model, clip, lora, strength_model, strength_clip):
|
||||||
key_map = {}
|
key_map = {}
|
||||||
if model is not None:
|
if model is not None:
|
||||||
@ -88,7 +61,7 @@ def load_lora_for_models(model, clip, lora, strength_model, strength_clip):
|
|||||||
|
|
||||||
|
|
||||||
class CLIP:
|
class CLIP:
|
||||||
def __init__(self, target=None, embedding_directory=None, no_init=False):
|
def __init__(self, target=None, embedding_directory=None, no_init=False, tokenizer_data={}):
|
||||||
if no_init:
|
if no_init:
|
||||||
return
|
return
|
||||||
params = target.params.copy()
|
params = target.params.copy()
|
||||||
@ -98,13 +71,19 @@ class CLIP:
|
|||||||
load_device = model_management.text_encoder_device()
|
load_device = model_management.text_encoder_device()
|
||||||
offload_device = model_management.text_encoder_offload_device()
|
offload_device = model_management.text_encoder_offload_device()
|
||||||
params['device'] = offload_device
|
params['device'] = offload_device
|
||||||
params['dtype'] = model_management.text_encoder_dtype(load_device)
|
dtype = model_management.text_encoder_dtype(load_device)
|
||||||
|
params['dtype'] = dtype
|
||||||
|
|
||||||
self.cond_stage_model = clip(**(params))
|
self.cond_stage_model = clip(**(params))
|
||||||
|
|
||||||
self.tokenizer = tokenizer(embedding_directory=embedding_directory)
|
for dt in self.cond_stage_model.dtypes:
|
||||||
|
if not model_management.supports_cast(load_device, dt):
|
||||||
|
load_device = offload_device
|
||||||
|
|
||||||
|
self.tokenizer = tokenizer(embedding_directory=embedding_directory, tokenizer_data=tokenizer_data)
|
||||||
self.patcher = comfy.model_patcher.ModelPatcher(self.cond_stage_model, load_device=load_device, offload_device=offload_device)
|
self.patcher = comfy.model_patcher.ModelPatcher(self.cond_stage_model, load_device=load_device, offload_device=offload_device)
|
||||||
self.layer_idx = None
|
self.layer_idx = None
|
||||||
|
logging.debug("CLIP model load device: {}, offload device: {}".format(load_device, offload_device))
|
||||||
|
|
||||||
def clone(self):
|
def clone(self):
|
||||||
n = CLIP(no_init=True)
|
n = CLIP(no_init=True)
|
||||||
@ -123,7 +102,7 @@ class CLIP:
|
|||||||
def tokenize(self, text, return_word_ids=False):
|
def tokenize(self, text, return_word_ids=False):
|
||||||
return self.tokenizer.tokenize_with_weights(text, return_word_ids)
|
return self.tokenizer.tokenize_with_weights(text, return_word_ids)
|
||||||
|
|
||||||
def encode_from_tokens(self, tokens, return_pooled=False):
|
def encode_from_tokens(self, tokens, return_pooled=False, return_dict=False):
|
||||||
self.cond_stage_model.reset_clip_options()
|
self.cond_stage_model.reset_clip_options()
|
||||||
|
|
||||||
if self.layer_idx is not None:
|
if self.layer_idx is not None:
|
||||||
@ -133,7 +112,15 @@ class CLIP:
|
|||||||
self.cond_stage_model.set_clip_options({"projected_pooled": False})
|
self.cond_stage_model.set_clip_options({"projected_pooled": False})
|
||||||
|
|
||||||
self.load_model()
|
self.load_model()
|
||||||
cond, pooled = self.cond_stage_model.encode_token_weights(tokens)
|
o = self.cond_stage_model.encode_token_weights(tokens)
|
||||||
|
cond, pooled = o[:2]
|
||||||
|
if return_dict:
|
||||||
|
out = {"cond": cond, "pooled_output": pooled}
|
||||||
|
if len(o) > 2:
|
||||||
|
for k in o[2]:
|
||||||
|
out[k] = o[2][k]
|
||||||
|
return out
|
||||||
|
|
||||||
if return_pooled:
|
if return_pooled:
|
||||||
return cond, pooled
|
return cond, pooled
|
||||||
return cond
|
return cond
|
||||||
@ -149,7 +136,11 @@ class CLIP:
|
|||||||
return self.cond_stage_model.load_sd(sd)
|
return self.cond_stage_model.load_sd(sd)
|
||||||
|
|
||||||
def get_sd(self):
|
def get_sd(self):
|
||||||
return self.cond_stage_model.state_dict()
|
sd_clip = self.cond_stage_model.state_dict()
|
||||||
|
sd_tokenizer = self.tokenizer.state_dict()
|
||||||
|
for k in sd_tokenizer:
|
||||||
|
sd_clip[k] = sd_tokenizer[k]
|
||||||
|
return sd_clip
|
||||||
|
|
||||||
def load_model(self):
|
def load_model(self):
|
||||||
model_management.load_model_gpu(self.patcher)
|
model_management.load_model_gpu(self.patcher)
|
||||||
@ -168,8 +159,10 @@ class VAE:
|
|||||||
self.downscale_ratio = 8
|
self.downscale_ratio = 8
|
||||||
self.upscale_ratio = 8
|
self.upscale_ratio = 8
|
||||||
self.latent_channels = 4
|
self.latent_channels = 4
|
||||||
|
self.output_channels = 3
|
||||||
self.process_input = lambda image: image * 2.0 - 1.0
|
self.process_input = lambda image: image * 2.0 - 1.0
|
||||||
self.process_output = lambda image: torch.clamp((image + 1.0) / 2.0, min=0.0, max=1.0)
|
self.process_output = lambda image: torch.clamp((image + 1.0) / 2.0, min=0.0, max=1.0)
|
||||||
|
self.working_dtypes = [torch.bfloat16, torch.float32]
|
||||||
|
|
||||||
if config is None:
|
if config is None:
|
||||||
if "decoder.mid.block_1.mix_factor" in sd:
|
if "decoder.mid.block_1.mix_factor" in sd:
|
||||||
@ -181,7 +174,8 @@ class VAE:
|
|||||||
encoder_config={'target': "comfy.ldm.modules.diffusionmodules.model.Encoder", 'params': encoder_config},
|
encoder_config={'target': "comfy.ldm.modules.diffusionmodules.model.Encoder", 'params': encoder_config},
|
||||||
decoder_config={'target': "comfy.ldm.modules.temporal_ae.VideoDecoder", 'params': decoder_config})
|
decoder_config={'target': "comfy.ldm.modules.temporal_ae.VideoDecoder", 'params': decoder_config})
|
||||||
elif "taesd_decoder.1.weight" in sd:
|
elif "taesd_decoder.1.weight" in sd:
|
||||||
self.first_stage_model = comfy.taesd.taesd.TAESD()
|
self.latent_channels = sd["taesd_decoder.1.weight"].shape[1]
|
||||||
|
self.first_stage_model = comfy.taesd.taesd.TAESD(latent_channels=self.latent_channels)
|
||||||
elif "vquantizer.codebook.weight" in sd: #VQGan: stage a of stable cascade
|
elif "vquantizer.codebook.weight" in sd: #VQGan: stage a of stable cascade
|
||||||
self.first_stage_model = StageA()
|
self.first_stage_model = StageA()
|
||||||
self.downscale_ratio = 4
|
self.downscale_ratio = 4
|
||||||
@ -226,6 +220,17 @@ class VAE:
|
|||||||
self.first_stage_model = AutoencodingEngine(regularizer_config={'target': "comfy.ldm.models.autoencoder.DiagonalGaussianRegularizer"},
|
self.first_stage_model = AutoencodingEngine(regularizer_config={'target': "comfy.ldm.models.autoencoder.DiagonalGaussianRegularizer"},
|
||||||
encoder_config={'target': "comfy.ldm.modules.diffusionmodules.model.Encoder", 'params': ddconfig},
|
encoder_config={'target': "comfy.ldm.modules.diffusionmodules.model.Encoder", 'params': ddconfig},
|
||||||
decoder_config={'target': "comfy.ldm.modules.diffusionmodules.model.Decoder", 'params': ddconfig})
|
decoder_config={'target': "comfy.ldm.modules.diffusionmodules.model.Decoder", 'params': ddconfig})
|
||||||
|
elif "decoder.layers.1.layers.0.beta" in sd:
|
||||||
|
self.first_stage_model = AudioOobleckVAE()
|
||||||
|
self.memory_used_encode = lambda shape, dtype: (1000 * shape[2]) * model_management.dtype_size(dtype)
|
||||||
|
self.memory_used_decode = lambda shape, dtype: (1000 * shape[2] * 2048) * model_management.dtype_size(dtype)
|
||||||
|
self.latent_channels = 64
|
||||||
|
self.output_channels = 2
|
||||||
|
self.upscale_ratio = 2048
|
||||||
|
self.downscale_ratio = 2048
|
||||||
|
self.process_output = lambda audio: audio
|
||||||
|
self.process_input = lambda audio: audio
|
||||||
|
self.working_dtypes = [torch.float16, torch.bfloat16, torch.float32]
|
||||||
else:
|
else:
|
||||||
logging.warning("WARNING: No VAE weights detected, VAE not initalized.")
|
logging.warning("WARNING: No VAE weights detected, VAE not initalized.")
|
||||||
self.first_stage_model = None
|
self.first_stage_model = None
|
||||||
@ -246,20 +251,21 @@ class VAE:
|
|||||||
self.device = device
|
self.device = device
|
||||||
offload_device = model_management.vae_offload_device()
|
offload_device = model_management.vae_offload_device()
|
||||||
if dtype is None:
|
if dtype is None:
|
||||||
dtype = model_management.vae_dtype()
|
dtype = model_management.vae_dtype(self.device, self.working_dtypes)
|
||||||
self.vae_dtype = dtype
|
self.vae_dtype = dtype
|
||||||
self.first_stage_model.to(self.vae_dtype)
|
self.first_stage_model.to(self.vae_dtype)
|
||||||
self.output_device = model_management.intermediate_device()
|
self.output_device = model_management.intermediate_device()
|
||||||
|
|
||||||
self.patcher = comfy.model_patcher.ModelPatcher(self.first_stage_model, load_device=self.device, offload_device=offload_device)
|
self.patcher = comfy.model_patcher.ModelPatcher(self.first_stage_model, load_device=self.device, offload_device=offload_device)
|
||||||
|
logging.debug("VAE load device: {}, offload device: {}, dtype: {}".format(self.device, offload_device, self.vae_dtype))
|
||||||
|
|
||||||
def vae_encode_crop_pixels(self, pixels):
|
def vae_encode_crop_pixels(self, pixels):
|
||||||
x = (pixels.shape[1] // self.downscale_ratio) * self.downscale_ratio
|
dims = pixels.shape[1:-1]
|
||||||
y = (pixels.shape[2] // self.downscale_ratio) * self.downscale_ratio
|
for d in range(len(dims)):
|
||||||
if pixels.shape[1] != x or pixels.shape[2] != y:
|
x = (dims[d] // self.downscale_ratio) * self.downscale_ratio
|
||||||
x_offset = (pixels.shape[1] % self.downscale_ratio) // 2
|
x_offset = (dims[d] % self.downscale_ratio) // 2
|
||||||
y_offset = (pixels.shape[2] % self.downscale_ratio) // 2
|
if x != dims[d]:
|
||||||
pixels = pixels[:, x_offset:x + x_offset, y_offset:y + y_offset, :]
|
pixels = pixels.narrow(d + 1, x_offset, x)
|
||||||
return pixels
|
return pixels
|
||||||
|
|
||||||
def decode_tiled_(self, samples, tile_x=64, tile_y=64, overlap = 16):
|
def decode_tiled_(self, samples, tile_x=64, tile_y=64, overlap = 16):
|
||||||
@ -276,6 +282,10 @@ class VAE:
|
|||||||
/ 3.0)
|
/ 3.0)
|
||||||
return output
|
return output
|
||||||
|
|
||||||
|
def decode_tiled_1d(self, samples, tile_x=128, overlap=32):
|
||||||
|
decode_fn = lambda a: self.first_stage_model.decode(a.to(self.vae_dtype).to(self.device)).float()
|
||||||
|
return comfy.utils.tiled_scale_multidim(samples, decode_fn, tile=(tile_x,), overlap=overlap, upscale_amount=self.upscale_ratio, out_channels=self.output_channels, output_device=self.output_device)
|
||||||
|
|
||||||
def encode_tiled_(self, pixel_samples, tile_x=512, tile_y=512, overlap = 64):
|
def encode_tiled_(self, pixel_samples, tile_x=512, tile_y=512, overlap = 64):
|
||||||
steps = pixel_samples.shape[0] * comfy.utils.get_tiled_scale_steps(pixel_samples.shape[3], pixel_samples.shape[2], tile_x, tile_y, overlap)
|
steps = pixel_samples.shape[0] * comfy.utils.get_tiled_scale_steps(pixel_samples.shape[3], pixel_samples.shape[2], tile_x, tile_y, overlap)
|
||||||
steps += pixel_samples.shape[0] * comfy.utils.get_tiled_scale_steps(pixel_samples.shape[3], pixel_samples.shape[2], tile_x // 2, tile_y * 2, overlap)
|
steps += pixel_samples.shape[0] * comfy.utils.get_tiled_scale_steps(pixel_samples.shape[3], pixel_samples.shape[2], tile_x // 2, tile_y * 2, overlap)
|
||||||
@ -289,6 +299,10 @@ class VAE:
|
|||||||
samples /= 3.0
|
samples /= 3.0
|
||||||
return samples
|
return samples
|
||||||
|
|
||||||
|
def encode_tiled_1d(self, samples, tile_x=128 * 2048, overlap=32 * 2048):
|
||||||
|
encode_fn = lambda a: self.first_stage_model.encode((self.process_input(a)).to(self.vae_dtype).to(self.device)).float()
|
||||||
|
return comfy.utils.tiled_scale_multidim(samples, encode_fn, tile=(tile_x,), overlap=overlap, upscale_amount=(1/self.downscale_ratio), out_channels=self.latent_channels, output_device=self.output_device)
|
||||||
|
|
||||||
def decode(self, samples_in):
|
def decode(self, samples_in):
|
||||||
try:
|
try:
|
||||||
memory_used = self.memory_used_decode(samples_in.shape, self.vae_dtype)
|
memory_used = self.memory_used_decode(samples_in.shape, self.vae_dtype)
|
||||||
@ -297,13 +311,16 @@ class VAE:
|
|||||||
batch_number = int(free_memory / memory_used)
|
batch_number = int(free_memory / memory_used)
|
||||||
batch_number = max(1, batch_number)
|
batch_number = max(1, batch_number)
|
||||||
|
|
||||||
pixel_samples = torch.empty((samples_in.shape[0], 3, round(samples_in.shape[2] * self.upscale_ratio), round(samples_in.shape[3] * self.upscale_ratio)), device=self.output_device)
|
pixel_samples = torch.empty((samples_in.shape[0], self.output_channels) + tuple(map(lambda a: a * self.upscale_ratio, samples_in.shape[2:])), device=self.output_device)
|
||||||
for x in range(0, samples_in.shape[0], batch_number):
|
for x in range(0, samples_in.shape[0], batch_number):
|
||||||
samples = samples_in[x:x+batch_number].to(self.vae_dtype).to(self.device)
|
samples = samples_in[x:x+batch_number].to(self.vae_dtype).to(self.device)
|
||||||
pixel_samples[x:x+batch_number] = self.process_output(self.first_stage_model.decode(samples).to(self.output_device).float())
|
pixel_samples[x:x+batch_number] = self.process_output(self.first_stage_model.decode(samples).to(self.output_device).float())
|
||||||
except model_management.OOM_EXCEPTION as e:
|
except model_management.OOM_EXCEPTION as e:
|
||||||
logging.warning("Warning: Ran out of memory when regular VAE decoding, retrying with tiled VAE decoding.")
|
logging.warning("Warning: Ran out of memory when regular VAE decoding, retrying with tiled VAE decoding.")
|
||||||
pixel_samples = self.decode_tiled_(samples_in)
|
if len(samples_in.shape) == 3:
|
||||||
|
pixel_samples = self.decode_tiled_1d(samples_in)
|
||||||
|
else:
|
||||||
|
pixel_samples = self.decode_tiled_(samples_in)
|
||||||
|
|
||||||
pixel_samples = pixel_samples.to(self.output_device).movedim(1,-1)
|
pixel_samples = pixel_samples.to(self.output_device).movedim(1,-1)
|
||||||
return pixel_samples
|
return pixel_samples
|
||||||
@ -322,14 +339,17 @@ class VAE:
|
|||||||
free_memory = model_management.get_free_memory(self.device)
|
free_memory = model_management.get_free_memory(self.device)
|
||||||
batch_number = int(free_memory / memory_used)
|
batch_number = int(free_memory / memory_used)
|
||||||
batch_number = max(1, batch_number)
|
batch_number = max(1, batch_number)
|
||||||
samples = torch.empty((pixel_samples.shape[0], self.latent_channels, round(pixel_samples.shape[2] // self.downscale_ratio), round(pixel_samples.shape[3] // self.downscale_ratio)), device=self.output_device)
|
samples = torch.empty((pixel_samples.shape[0], self.latent_channels) + tuple(map(lambda a: a // self.downscale_ratio, pixel_samples.shape[2:])), device=self.output_device)
|
||||||
for x in range(0, pixel_samples.shape[0], batch_number):
|
for x in range(0, pixel_samples.shape[0], batch_number):
|
||||||
pixels_in = self.process_input(pixel_samples[x:x+batch_number]).to(self.vae_dtype).to(self.device)
|
pixels_in = self.process_input(pixel_samples[x:x+batch_number]).to(self.vae_dtype).to(self.device)
|
||||||
samples[x:x+batch_number] = self.first_stage_model.encode(pixels_in).to(self.output_device).float()
|
samples[x:x+batch_number] = self.first_stage_model.encode(pixels_in).to(self.output_device).float()
|
||||||
|
|
||||||
except model_management.OOM_EXCEPTION as e:
|
except model_management.OOM_EXCEPTION as e:
|
||||||
logging.warning("Warning: Ran out of memory when regular VAE encoding, retrying with tiled VAE encoding.")
|
logging.warning("Warning: Ran out of memory when regular VAE encoding, retrying with tiled VAE encoding.")
|
||||||
samples = self.encode_tiled_(pixel_samples)
|
if len(pixel_samples.shape) == 3:
|
||||||
|
samples = self.encode_tiled_1d(pixel_samples)
|
||||||
|
else:
|
||||||
|
samples = self.encode_tiled_(pixel_samples)
|
||||||
|
|
||||||
return samples
|
return samples
|
||||||
|
|
||||||
@ -364,6 +384,9 @@ def load_style_model(ckpt_path):
|
|||||||
class CLIPType(Enum):
|
class CLIPType(Enum):
|
||||||
STABLE_DIFFUSION = 1
|
STABLE_DIFFUSION = 1
|
||||||
STABLE_CASCADE = 2
|
STABLE_CASCADE = 2
|
||||||
|
SD3 = 3
|
||||||
|
STABLE_AUDIO = 4
|
||||||
|
HUNYUAN_DIT = 5
|
||||||
|
|
||||||
def load_clip(ckpt_paths, embedding_directory=None, clip_type=CLIPType.STABLE_DIFFUSION):
|
def load_clip(ckpt_paths, embedding_directory=None, clip_type=CLIPType.STABLE_DIFFUSION):
|
||||||
clip_data = []
|
clip_data = []
|
||||||
@ -393,12 +416,34 @@ def load_clip(ckpt_paths, embedding_directory=None, clip_type=CLIPType.STABLE_DI
|
|||||||
elif "text_model.encoder.layers.22.mlp.fc1.weight" in clip_data[0]:
|
elif "text_model.encoder.layers.22.mlp.fc1.weight" in clip_data[0]:
|
||||||
clip_target.clip = sd2_clip.SD2ClipModel
|
clip_target.clip = sd2_clip.SD2ClipModel
|
||||||
clip_target.tokenizer = sd2_clip.SD2Tokenizer
|
clip_target.tokenizer = sd2_clip.SD2Tokenizer
|
||||||
|
elif "encoder.block.23.layer.1.DenseReluDense.wi_1.weight" in clip_data[0]:
|
||||||
|
weight = clip_data[0]["encoder.block.23.layer.1.DenseReluDense.wi_1.weight"]
|
||||||
|
dtype_t5 = weight.dtype
|
||||||
|
if weight.shape[-1] == 4096:
|
||||||
|
clip_target.clip = comfy.text_encoders.sd3_clip.sd3_clip(clip_l=False, clip_g=False, t5=True, dtype_t5=dtype_t5)
|
||||||
|
clip_target.tokenizer = comfy.text_encoders.sd3_clip.SD3Tokenizer
|
||||||
|
elif weight.shape[-1] == 2048:
|
||||||
|
clip_target.clip = comfy.text_encoders.aura_t5.AuraT5Model
|
||||||
|
clip_target.tokenizer = comfy.text_encoders.aura_t5.AuraT5Tokenizer
|
||||||
|
elif "encoder.block.0.layer.0.SelfAttention.k.weight" in clip_data[0]:
|
||||||
|
clip_target.clip = comfy.text_encoders.sa_t5.SAT5Model
|
||||||
|
clip_target.tokenizer = comfy.text_encoders.sa_t5.SAT5Tokenizer
|
||||||
else:
|
else:
|
||||||
clip_target.clip = sd1_clip.SD1ClipModel
|
clip_target.clip = sd1_clip.SD1ClipModel
|
||||||
clip_target.tokenizer = sd1_clip.SD1Tokenizer
|
clip_target.tokenizer = sd1_clip.SD1Tokenizer
|
||||||
else:
|
elif len(clip_data) == 2:
|
||||||
clip_target.clip = sdxl_clip.SDXLClipModel
|
if clip_type == CLIPType.SD3:
|
||||||
clip_target.tokenizer = sdxl_clip.SDXLTokenizer
|
clip_target.clip = comfy.text_encoders.sd3_clip.sd3_clip(clip_l=True, clip_g=True, t5=False)
|
||||||
|
clip_target.tokenizer = comfy.text_encoders.sd3_clip.SD3Tokenizer
|
||||||
|
elif clip_type == CLIPType.HUNYUAN_DIT:
|
||||||
|
clip_target.clip = comfy.text_encoders.hydit.HyditModel
|
||||||
|
clip_target.tokenizer = comfy.text_encoders.hydit.HyditTokenizer
|
||||||
|
else:
|
||||||
|
clip_target.clip = sdxl_clip.SDXLClipModel
|
||||||
|
clip_target.tokenizer = sdxl_clip.SDXLTokenizer
|
||||||
|
elif len(clip_data) == 3:
|
||||||
|
clip_target.clip = comfy.text_encoders.sd3_clip.SD3ClipModel
|
||||||
|
clip_target.tokenizer = comfy.text_encoders.sd3_clip.SD3Tokenizer
|
||||||
|
|
||||||
clip = CLIP(clip_target, embedding_directory=embedding_directory)
|
clip = CLIP(clip_target, embedding_directory=embedding_directory)
|
||||||
for c in clip_data:
|
for c in clip_data:
|
||||||
@ -452,17 +497,18 @@ def load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, o
|
|||||||
model_patcher = None
|
model_patcher = None
|
||||||
clip_target = None
|
clip_target = None
|
||||||
|
|
||||||
parameters = comfy.utils.calculate_parameters(sd, "model.diffusion_model.")
|
diffusion_model_prefix = model_detection.unet_prefix_from_state_dict(sd)
|
||||||
|
parameters = comfy.utils.calculate_parameters(sd, diffusion_model_prefix)
|
||||||
load_device = model_management.get_torch_device()
|
load_device = model_management.get_torch_device()
|
||||||
|
|
||||||
model_config = model_detection.model_config_from_unet(sd, "model.diffusion_model.")
|
model_config = model_detection.model_config_from_unet(sd, diffusion_model_prefix)
|
||||||
|
if model_config is None:
|
||||||
|
raise RuntimeError("ERROR: Could not detect model type of: {}".format(ckpt_path))
|
||||||
|
|
||||||
unet_dtype = model_management.unet_dtype(model_params=parameters, supported_dtypes=model_config.supported_inference_dtypes)
|
unet_dtype = model_management.unet_dtype(model_params=parameters, supported_dtypes=model_config.supported_inference_dtypes)
|
||||||
manual_cast_dtype = model_management.unet_manual_cast(unet_dtype, load_device, model_config.supported_inference_dtypes)
|
manual_cast_dtype = model_management.unet_manual_cast(unet_dtype, load_device, model_config.supported_inference_dtypes)
|
||||||
model_config.set_inference_dtype(unet_dtype, manual_cast_dtype)
|
model_config.set_inference_dtype(unet_dtype, manual_cast_dtype)
|
||||||
|
|
||||||
if model_config is None:
|
|
||||||
raise RuntimeError("ERROR: Could not detect model type of: {}".format(ckpt_path))
|
|
||||||
|
|
||||||
if model_config.clip_vision_prefix is not None:
|
if model_config.clip_vision_prefix is not None:
|
||||||
if output_clipvision:
|
if output_clipvision:
|
||||||
clipvision = clip_vision.load_clipvision_from_sd(sd, model_config.clip_vision_prefix, True)
|
clipvision = clip_vision.load_clipvision_from_sd(sd, model_config.clip_vision_prefix, True)
|
||||||
@ -470,8 +516,8 @@ def load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, o
|
|||||||
if output_model:
|
if output_model:
|
||||||
inital_load_device = model_management.unet_inital_load_device(parameters, unet_dtype)
|
inital_load_device = model_management.unet_inital_load_device(parameters, unet_dtype)
|
||||||
offload_device = model_management.unet_offload_device()
|
offload_device = model_management.unet_offload_device()
|
||||||
model = model_config.get_model(sd, "model.diffusion_model.", device=inital_load_device)
|
model = model_config.get_model(sd, diffusion_model_prefix, device=inital_load_device)
|
||||||
model.load_model_weights(sd, "model.diffusion_model.")
|
model.load_model_weights(sd, diffusion_model_prefix)
|
||||||
|
|
||||||
if output_vae:
|
if output_vae:
|
||||||
vae_sd = comfy.utils.state_dict_prefix_replace(sd, {k: "" for k in model_config.vae_key_prefix}, filter_keys=True)
|
vae_sd = comfy.utils.state_dict_prefix_replace(sd, {k: "" for k in model_config.vae_key_prefix}, filter_keys=True)
|
||||||
@ -479,11 +525,11 @@ def load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, o
|
|||||||
vae = VAE(sd=vae_sd)
|
vae = VAE(sd=vae_sd)
|
||||||
|
|
||||||
if output_clip:
|
if output_clip:
|
||||||
clip_target = model_config.clip_target()
|
clip_target = model_config.clip_target(state_dict=sd)
|
||||||
if clip_target is not None:
|
if clip_target is not None:
|
||||||
clip_sd = model_config.process_clip_state_dict(sd)
|
clip_sd = model_config.process_clip_state_dict(sd)
|
||||||
if len(clip_sd) > 0:
|
if len(clip_sd) > 0:
|
||||||
clip = CLIP(clip_target, embedding_directory=embedding_directory)
|
clip = CLIP(clip_target, embedding_directory=embedding_directory, tokenizer_data=clip_sd)
|
||||||
m, u = clip.load_sd(clip_sd, full_model=True)
|
m, u = clip.load_sd(clip_sd, full_model=True)
|
||||||
if len(m) > 0:
|
if len(m) > 0:
|
||||||
m_filter = list(filter(lambda a: ".logit_scale" not in a and ".transformer.text_projection.weight" not in a, m))
|
m_filter = list(filter(lambda a: ".logit_scale" not in a and ".transformer.text_projection.weight" not in a, m))
|
||||||
@ -510,30 +556,40 @@ def load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, o
|
|||||||
return (model_patcher, clip, vae, clipvision)
|
return (model_patcher, clip, vae, clipvision)
|
||||||
|
|
||||||
|
|
||||||
def load_unet_state_dict(sd): #load unet in diffusers format
|
def load_unet_state_dict(sd): #load unet in diffusers or regular format
|
||||||
|
|
||||||
|
#Allow loading unets from checkpoint files
|
||||||
|
diffusion_model_prefix = model_detection.unet_prefix_from_state_dict(sd)
|
||||||
|
temp_sd = comfy.utils.state_dict_prefix_replace(sd, {diffusion_model_prefix: ""}, filter_keys=True)
|
||||||
|
if len(temp_sd) > 0:
|
||||||
|
sd = temp_sd
|
||||||
|
|
||||||
parameters = comfy.utils.calculate_parameters(sd)
|
parameters = comfy.utils.calculate_parameters(sd)
|
||||||
unet_dtype = model_management.unet_dtype(model_params=parameters)
|
unet_dtype = model_management.unet_dtype(model_params=parameters)
|
||||||
load_device = model_management.get_torch_device()
|
load_device = model_management.get_torch_device()
|
||||||
|
model_config = model_detection.model_config_from_unet(sd, "")
|
||||||
|
|
||||||
if "input_blocks.0.0.weight" in sd or 'clf.1.weight' in sd: #ldm or stable cascade
|
if model_config is not None:
|
||||||
model_config = model_detection.model_config_from_unet(sd, "")
|
|
||||||
if model_config is None:
|
|
||||||
return None
|
|
||||||
new_sd = sd
|
new_sd = sd
|
||||||
|
else:
|
||||||
|
new_sd = model_detection.convert_diffusers_mmdit(sd, "")
|
||||||
|
if new_sd is not None: #diffusers mmdit
|
||||||
|
model_config = model_detection.model_config_from_unet(new_sd, "")
|
||||||
|
if model_config is None:
|
||||||
|
return None
|
||||||
|
else: #diffusers unet
|
||||||
|
model_config = model_detection.model_config_from_diffusers_unet(sd)
|
||||||
|
if model_config is None:
|
||||||
|
return None
|
||||||
|
|
||||||
else: #diffusers
|
diffusers_keys = comfy.utils.unet_to_diffusers(model_config.unet_config)
|
||||||
model_config = model_detection.model_config_from_diffusers_unet(sd)
|
|
||||||
if model_config is None:
|
|
||||||
return None
|
|
||||||
|
|
||||||
diffusers_keys = comfy.utils.unet_to_diffusers(model_config.unet_config)
|
new_sd = {}
|
||||||
|
for k in diffusers_keys:
|
||||||
new_sd = {}
|
if k in sd:
|
||||||
for k in diffusers_keys:
|
new_sd[diffusers_keys[k]] = sd.pop(k)
|
||||||
if k in sd:
|
else:
|
||||||
new_sd[diffusers_keys[k]] = sd.pop(k)
|
logging.warning("{} {}".format(diffusers_keys[k], k))
|
||||||
else:
|
|
||||||
logging.warning("{} {}".format(diffusers_keys[k], k))
|
|
||||||
|
|
||||||
offload_device = model_management.unet_offload_device()
|
offload_device = model_management.unet_offload_device()
|
||||||
unet_dtype = model_management.unet_dtype(model_params=parameters, supported_dtypes=model_config.supported_inference_dtypes)
|
unet_dtype = model_management.unet_dtype(model_params=parameters, supported_dtypes=model_config.supported_inference_dtypes)
|
||||||
@ -568,4 +624,9 @@ def save_checkpoint(output_path, model, clip=None, vae=None, clip_vision=None, m
|
|||||||
for k in extra_keys:
|
for k in extra_keys:
|
||||||
sd[k] = extra_keys[k]
|
sd[k] = extra_keys[k]
|
||||||
|
|
||||||
|
for k in sd:
|
||||||
|
t = sd[k]
|
||||||
|
if not t.is_contiguous():
|
||||||
|
sd[k] = t.contiguous()
|
||||||
|
|
||||||
comfy.utils.save_torch_file(sd, output_path, metadata=metadata)
|
comfy.utils.save_torch_file(sd, output_path, metadata=metadata)
|
||||||
|
|||||||
@ -9,6 +9,7 @@ from . import model_management
|
|||||||
import comfy.clip_model
|
import comfy.clip_model
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
import numbers
|
||||||
|
|
||||||
def gen_empty_tokens(special_tokens, length):
|
def gen_empty_tokens(special_tokens, length):
|
||||||
start_token = special_tokens.get("start", None)
|
start_token = special_tokens.get("start", None)
|
||||||
@ -37,7 +38,9 @@ class ClipTokenWeightEncoder:
|
|||||||
if has_weights or sections == 0:
|
if has_weights or sections == 0:
|
||||||
to_encode.append(gen_empty_tokens(self.special_tokens, max_token_len))
|
to_encode.append(gen_empty_tokens(self.special_tokens, max_token_len))
|
||||||
|
|
||||||
out, pooled = self.encode(to_encode)
|
o = self.encode(to_encode)
|
||||||
|
out, pooled = o[:2]
|
||||||
|
|
||||||
if pooled is not None:
|
if pooled is not None:
|
||||||
first_pooled = pooled[0:1].to(model_management.intermediate_device())
|
first_pooled = pooled[0:1].to(model_management.intermediate_device())
|
||||||
else:
|
else:
|
||||||
@ -56,8 +59,20 @@ class ClipTokenWeightEncoder:
|
|||||||
output.append(z)
|
output.append(z)
|
||||||
|
|
||||||
if (len(output) == 0):
|
if (len(output) == 0):
|
||||||
return out[-1:].to(model_management.intermediate_device()), first_pooled
|
r = (out[-1:].to(model_management.intermediate_device()), first_pooled)
|
||||||
return torch.cat(output, dim=-2).to(model_management.intermediate_device()), first_pooled
|
else:
|
||||||
|
r = (torch.cat(output, dim=-2).to(model_management.intermediate_device()), first_pooled)
|
||||||
|
|
||||||
|
if len(o) > 2:
|
||||||
|
extra = {}
|
||||||
|
for k in o[2]:
|
||||||
|
v = o[2][k]
|
||||||
|
if k == "attention_mask":
|
||||||
|
v = v[:sections].flatten().unsqueeze(dim=0).to(model_management.intermediate_device())
|
||||||
|
extra[k] = v
|
||||||
|
|
||||||
|
r = r + (extra,)
|
||||||
|
return r
|
||||||
|
|
||||||
class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
||||||
"""Uses the CLIP transformer encoder for text (from huggingface)"""
|
"""Uses the CLIP transformer encoder for text (from huggingface)"""
|
||||||
@ -68,7 +83,8 @@ class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
|||||||
]
|
]
|
||||||
def __init__(self, version="openai/clip-vit-large-patch14", device="cpu", max_length=77,
|
def __init__(self, version="openai/clip-vit-large-patch14", device="cpu", max_length=77,
|
||||||
freeze=True, layer="last", layer_idx=None, textmodel_json_config=None, dtype=None, model_class=comfy.clip_model.CLIPTextModel,
|
freeze=True, layer="last", layer_idx=None, textmodel_json_config=None, dtype=None, model_class=comfy.clip_model.CLIPTextModel,
|
||||||
special_tokens={"start": 49406, "end": 49407, "pad": 49407}, layer_norm_hidden_state=True, enable_attention_masks=False, return_projected_pooled=True): # clip-vit-base-patch32
|
special_tokens={"start": 49406, "end": 49407, "pad": 49407}, layer_norm_hidden_state=True, enable_attention_masks=False, zero_out_masked=False,
|
||||||
|
return_projected_pooled=True, return_attention_masks=False): # clip-vit-base-patch32
|
||||||
super().__init__()
|
super().__init__()
|
||||||
assert layer in self.LAYERS
|
assert layer in self.LAYERS
|
||||||
|
|
||||||
@ -90,9 +106,11 @@ class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
|||||||
|
|
||||||
self.logit_scale = torch.nn.Parameter(torch.tensor(4.6055))
|
self.logit_scale = torch.nn.Parameter(torch.tensor(4.6055))
|
||||||
self.enable_attention_masks = enable_attention_masks
|
self.enable_attention_masks = enable_attention_masks
|
||||||
|
self.zero_out_masked = zero_out_masked
|
||||||
|
|
||||||
self.layer_norm_hidden_state = layer_norm_hidden_state
|
self.layer_norm_hidden_state = layer_norm_hidden_state
|
||||||
self.return_projected_pooled = return_projected_pooled
|
self.return_projected_pooled = return_projected_pooled
|
||||||
|
self.return_attention_masks = return_attention_masks
|
||||||
|
|
||||||
if layer == "hidden":
|
if layer == "hidden":
|
||||||
assert layer_idx is not None
|
assert layer_idx is not None
|
||||||
@ -128,10 +146,10 @@ class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
|||||||
for x in tokens:
|
for x in tokens:
|
||||||
tokens_temp = []
|
tokens_temp = []
|
||||||
for y in x:
|
for y in x:
|
||||||
if isinstance(y, int):
|
if isinstance(y, numbers.Integral):
|
||||||
if y == token_dict_size: #EOS token
|
if y == token_dict_size: #EOS token
|
||||||
y = -1
|
y = -1
|
||||||
tokens_temp += [y]
|
tokens_temp += [int(y)]
|
||||||
else:
|
else:
|
||||||
if y.shape[0] == current_embeds.weight.shape[1]:
|
if y.shape[0] == current_embeds.weight.shape[1]:
|
||||||
embedding_weights += [y]
|
embedding_weights += [y]
|
||||||
@ -166,22 +184,29 @@ class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
|||||||
tokens = torch.LongTensor(tokens).to(device)
|
tokens = torch.LongTensor(tokens).to(device)
|
||||||
|
|
||||||
attention_mask = None
|
attention_mask = None
|
||||||
if self.enable_attention_masks:
|
if self.enable_attention_masks or self.zero_out_masked or self.return_attention_masks:
|
||||||
attention_mask = torch.zeros_like(tokens)
|
attention_mask = torch.zeros_like(tokens)
|
||||||
max_token = self.transformer.get_input_embeddings().weight.shape[0] - 1
|
end_token = self.special_tokens.get("end", -1)
|
||||||
for x in range(attention_mask.shape[0]):
|
for x in range(attention_mask.shape[0]):
|
||||||
for y in range(attention_mask.shape[1]):
|
for y in range(attention_mask.shape[1]):
|
||||||
attention_mask[x, y] = 1
|
attention_mask[x, y] = 1
|
||||||
if tokens[x, y] == max_token:
|
if tokens[x, y] == end_token:
|
||||||
break
|
break
|
||||||
|
|
||||||
outputs = self.transformer(tokens, attention_mask, intermediate_output=self.layer_idx, final_layer_norm_intermediate=self.layer_norm_hidden_state)
|
attention_mask_model = None
|
||||||
|
if self.enable_attention_masks:
|
||||||
|
attention_mask_model = attention_mask
|
||||||
|
|
||||||
|
outputs = self.transformer(tokens, attention_mask_model, intermediate_output=self.layer_idx, final_layer_norm_intermediate=self.layer_norm_hidden_state)
|
||||||
self.transformer.set_input_embeddings(backup_embeds)
|
self.transformer.set_input_embeddings(backup_embeds)
|
||||||
|
|
||||||
if self.layer == "last":
|
if self.layer == "last":
|
||||||
z = outputs[0]
|
z = outputs[0].float()
|
||||||
else:
|
else:
|
||||||
z = outputs[1]
|
z = outputs[1].float()
|
||||||
|
|
||||||
|
if self.zero_out_masked:
|
||||||
|
z *= attention_mask.unsqueeze(-1).float()
|
||||||
|
|
||||||
pooled_output = None
|
pooled_output = None
|
||||||
if len(outputs) >= 3:
|
if len(outputs) >= 3:
|
||||||
@ -190,7 +215,14 @@ class SDClipModel(torch.nn.Module, ClipTokenWeightEncoder):
|
|||||||
elif outputs[2] is not None:
|
elif outputs[2] is not None:
|
||||||
pooled_output = outputs[2].float()
|
pooled_output = outputs[2].float()
|
||||||
|
|
||||||
return z.float(), pooled_output
|
extra = {}
|
||||||
|
if self.return_attention_masks:
|
||||||
|
extra["attention_mask"] = attention_mask
|
||||||
|
|
||||||
|
if len(extra) > 0:
|
||||||
|
return z, pooled_output, extra
|
||||||
|
|
||||||
|
return z, pooled_output
|
||||||
|
|
||||||
def encode(self, tokens):
|
def encode(self, tokens):
|
||||||
return self(tokens)
|
return self(tokens)
|
||||||
@ -354,7 +386,7 @@ def load_embed(embedding_name, embedding_directory, embedding_size, embed_key=No
|
|||||||
return embed_out
|
return embed_out
|
||||||
|
|
||||||
class SDTokenizer:
|
class SDTokenizer:
|
||||||
def __init__(self, tokenizer_path=None, max_length=77, pad_with_end=True, embedding_directory=None, embedding_size=768, embedding_key='clip_l', tokenizer_class=CLIPTokenizer, has_start_token=True, pad_to_max_length=True, min_length=None):
|
def __init__(self, tokenizer_path=None, max_length=77, pad_with_end=True, embedding_directory=None, embedding_size=768, embedding_key='clip_l', tokenizer_class=CLIPTokenizer, has_start_token=True, pad_to_max_length=True, min_length=None, pad_token=None, tokenizer_data={}):
|
||||||
if tokenizer_path is None:
|
if tokenizer_path is None:
|
||||||
tokenizer_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "sd1_tokenizer")
|
tokenizer_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "sd1_tokenizer")
|
||||||
self.tokenizer = tokenizer_class.from_pretrained(tokenizer_path)
|
self.tokenizer = tokenizer_class.from_pretrained(tokenizer_path)
|
||||||
@ -370,6 +402,14 @@ class SDTokenizer:
|
|||||||
self.tokens_start = 0
|
self.tokens_start = 0
|
||||||
self.start_token = None
|
self.start_token = None
|
||||||
self.end_token = empty[0]
|
self.end_token = empty[0]
|
||||||
|
|
||||||
|
if pad_token is not None:
|
||||||
|
self.pad_token = pad_token
|
||||||
|
elif pad_with_end:
|
||||||
|
self.pad_token = self.end_token
|
||||||
|
else:
|
||||||
|
self.pad_token = 0
|
||||||
|
|
||||||
self.pad_with_end = pad_with_end
|
self.pad_with_end = pad_with_end
|
||||||
self.pad_to_max_length = pad_to_max_length
|
self.pad_to_max_length = pad_to_max_length
|
||||||
|
|
||||||
@ -402,10 +442,6 @@ class SDTokenizer:
|
|||||||
Word id values are unique per word and embedding, where the id 0 is reserved for non word tokens.
|
Word id values are unique per word and embedding, where the id 0 is reserved for non word tokens.
|
||||||
Returned list has the dimensions NxM where M is the input size of CLIP
|
Returned list has the dimensions NxM where M is the input size of CLIP
|
||||||
'''
|
'''
|
||||||
if self.pad_with_end:
|
|
||||||
pad_token = self.end_token
|
|
||||||
else:
|
|
||||||
pad_token = 0
|
|
||||||
|
|
||||||
text = escape_important(text)
|
text = escape_important(text)
|
||||||
parsed_weights = token_weights(text, 1.0)
|
parsed_weights = token_weights(text, 1.0)
|
||||||
@ -457,7 +493,7 @@ class SDTokenizer:
|
|||||||
else:
|
else:
|
||||||
batch.append((self.end_token, 1.0, 0))
|
batch.append((self.end_token, 1.0, 0))
|
||||||
if self.pad_to_max_length:
|
if self.pad_to_max_length:
|
||||||
batch.extend([(pad_token, 1.0, 0)] * (remaining_length))
|
batch.extend([(self.pad_token, 1.0, 0)] * (remaining_length))
|
||||||
#start new batch
|
#start new batch
|
||||||
batch = []
|
batch = []
|
||||||
if self.start_token is not None:
|
if self.start_token is not None:
|
||||||
@ -470,9 +506,9 @@ class SDTokenizer:
|
|||||||
#fill last batch
|
#fill last batch
|
||||||
batch.append((self.end_token, 1.0, 0))
|
batch.append((self.end_token, 1.0, 0))
|
||||||
if self.pad_to_max_length:
|
if self.pad_to_max_length:
|
||||||
batch.extend([(pad_token, 1.0, 0)] * (self.max_length - len(batch)))
|
batch.extend([(self.pad_token, 1.0, 0)] * (self.max_length - len(batch)))
|
||||||
if self.min_length is not None and len(batch) < self.min_length:
|
if self.min_length is not None and len(batch) < self.min_length:
|
||||||
batch.extend([(pad_token, 1.0, 0)] * (self.min_length - len(batch)))
|
batch.extend([(self.pad_token, 1.0, 0)] * (self.min_length - len(batch)))
|
||||||
|
|
||||||
if not return_word_ids:
|
if not return_word_ids:
|
||||||
batched_tokens = [[(t, w) for t, w,_ in x] for x in batched_tokens]
|
batched_tokens = [[(t, w) for t, w,_ in x] for x in batched_tokens]
|
||||||
@ -483,12 +519,14 @@ class SDTokenizer:
|
|||||||
def untokenize(self, token_weight_pair):
|
def untokenize(self, token_weight_pair):
|
||||||
return list(map(lambda a: (a, self.inv_vocab[a[0]]), token_weight_pair))
|
return list(map(lambda a: (a, self.inv_vocab[a[0]]), token_weight_pair))
|
||||||
|
|
||||||
|
def state_dict(self):
|
||||||
|
return {}
|
||||||
|
|
||||||
class SD1Tokenizer:
|
class SD1Tokenizer:
|
||||||
def __init__(self, embedding_directory=None, clip_name="l", tokenizer=SDTokenizer):
|
def __init__(self, embedding_directory=None, tokenizer_data={}, clip_name="l", tokenizer=SDTokenizer):
|
||||||
self.clip_name = clip_name
|
self.clip_name = clip_name
|
||||||
self.clip = "clip_{}".format(self.clip_name)
|
self.clip = "clip_{}".format(self.clip_name)
|
||||||
setattr(self, self.clip, tokenizer(embedding_directory=embedding_directory))
|
setattr(self, self.clip, tokenizer(embedding_directory=embedding_directory, tokenizer_data=tokenizer_data))
|
||||||
|
|
||||||
def tokenize_with_weights(self, text:str, return_word_ids=False):
|
def tokenize_with_weights(self, text:str, return_word_ids=False):
|
||||||
out = {}
|
out = {}
|
||||||
@ -498,14 +536,26 @@ class SD1Tokenizer:
|
|||||||
def untokenize(self, token_weight_pair):
|
def untokenize(self, token_weight_pair):
|
||||||
return getattr(self, self.clip).untokenize(token_weight_pair)
|
return getattr(self, self.clip).untokenize(token_weight_pair)
|
||||||
|
|
||||||
|
def state_dict(self):
|
||||||
|
return {}
|
||||||
|
|
||||||
class SD1ClipModel(torch.nn.Module):
|
class SD1ClipModel(torch.nn.Module):
|
||||||
def __init__(self, device="cpu", dtype=None, clip_name="l", clip_model=SDClipModel, **kwargs):
|
def __init__(self, device="cpu", dtype=None, clip_name="l", clip_model=SDClipModel, name=None, **kwargs):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.clip_name = clip_name
|
|
||||||
self.clip = "clip_{}".format(self.clip_name)
|
if name is not None:
|
||||||
|
self.clip_name = name
|
||||||
|
self.clip = "{}".format(self.clip_name)
|
||||||
|
else:
|
||||||
|
self.clip_name = clip_name
|
||||||
|
self.clip = "clip_{}".format(self.clip_name)
|
||||||
|
|
||||||
setattr(self, self.clip, clip_model(device=device, dtype=dtype, **kwargs))
|
setattr(self, self.clip, clip_model(device=device, dtype=dtype, **kwargs))
|
||||||
|
|
||||||
|
self.dtypes = set()
|
||||||
|
if dtype is not None:
|
||||||
|
self.dtypes.add(dtype)
|
||||||
|
|
||||||
def set_clip_options(self, options):
|
def set_clip_options(self, options):
|
||||||
getattr(self, self.clip).set_clip_options(options)
|
getattr(self, self.clip).set_clip_options(options)
|
||||||
|
|
||||||
@ -514,8 +564,8 @@ class SD1ClipModel(torch.nn.Module):
|
|||||||
|
|
||||||
def encode_token_weights(self, token_weight_pairs):
|
def encode_token_weights(self, token_weight_pairs):
|
||||||
token_weight_pairs = token_weight_pairs[self.clip_name]
|
token_weight_pairs = token_weight_pairs[self.clip_name]
|
||||||
out, pooled = getattr(self, self.clip).encode_token_weights(token_weight_pairs)
|
out = getattr(self, self.clip).encode_token_weights(token_weight_pairs)
|
||||||
return out, pooled
|
return out
|
||||||
|
|
||||||
def load_sd(self, sd):
|
def load_sd(self, sd):
|
||||||
return getattr(self, self.clip).load_sd(sd)
|
return getattr(self, self.clip).load_sd(sd)
|
||||||
|
|||||||
@ -1,5 +1,4 @@
|
|||||||
from comfy import sd1_clip
|
from comfy import sd1_clip
|
||||||
import torch
|
|
||||||
import os
|
import os
|
||||||
|
|
||||||
class SD2ClipHModel(sd1_clip.SDClipModel):
|
class SD2ClipHModel(sd1_clip.SDClipModel):
|
||||||
@ -12,12 +11,12 @@ class SD2ClipHModel(sd1_clip.SDClipModel):
|
|||||||
super().__init__(device=device, freeze=freeze, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"start": 49406, "end": 49407, "pad": 0})
|
super().__init__(device=device, freeze=freeze, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"start": 49406, "end": 49407, "pad": 0})
|
||||||
|
|
||||||
class SD2ClipHTokenizer(sd1_clip.SDTokenizer):
|
class SD2ClipHTokenizer(sd1_clip.SDTokenizer):
|
||||||
def __init__(self, tokenizer_path=None, embedding_directory=None):
|
def __init__(self, tokenizer_path=None, embedding_directory=None, tokenizer_data={}):
|
||||||
super().__init__(tokenizer_path, pad_with_end=False, embedding_directory=embedding_directory, embedding_size=1024)
|
super().__init__(tokenizer_path, pad_with_end=False, embedding_directory=embedding_directory, embedding_size=1024)
|
||||||
|
|
||||||
class SD2Tokenizer(sd1_clip.SD1Tokenizer):
|
class SD2Tokenizer(sd1_clip.SD1Tokenizer):
|
||||||
def __init__(self, embedding_directory=None):
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
super().__init__(embedding_directory=embedding_directory, clip_name="h", tokenizer=SD2ClipHTokenizer)
|
super().__init__(embedding_directory=embedding_directory, tokenizer_data=tokenizer_data, clip_name="h", tokenizer=SD2ClipHTokenizer)
|
||||||
|
|
||||||
class SD2ClipModel(sd1_clip.SD1ClipModel):
|
class SD2ClipModel(sd1_clip.SD1ClipModel):
|
||||||
def __init__(self, device="cpu", dtype=None, **kwargs):
|
def __init__(self, device="cpu", dtype=None, **kwargs):
|
||||||
|
|||||||
@ -16,12 +16,12 @@ class SDXLClipG(sd1_clip.SDClipModel):
|
|||||||
return super().load_sd(sd)
|
return super().load_sd(sd)
|
||||||
|
|
||||||
class SDXLClipGTokenizer(sd1_clip.SDTokenizer):
|
class SDXLClipGTokenizer(sd1_clip.SDTokenizer):
|
||||||
def __init__(self, tokenizer_path=None, embedding_directory=None):
|
def __init__(self, tokenizer_path=None, embedding_directory=None, tokenizer_data={}):
|
||||||
super().__init__(tokenizer_path, pad_with_end=False, embedding_directory=embedding_directory, embedding_size=1280, embedding_key='clip_g')
|
super().__init__(tokenizer_path, pad_with_end=False, embedding_directory=embedding_directory, embedding_size=1280, embedding_key='clip_g')
|
||||||
|
|
||||||
|
|
||||||
class SDXLTokenizer:
|
class SDXLTokenizer:
|
||||||
def __init__(self, embedding_directory=None):
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
self.clip_l = sd1_clip.SDTokenizer(embedding_directory=embedding_directory)
|
self.clip_l = sd1_clip.SDTokenizer(embedding_directory=embedding_directory)
|
||||||
self.clip_g = SDXLClipGTokenizer(embedding_directory=embedding_directory)
|
self.clip_g = SDXLClipGTokenizer(embedding_directory=embedding_directory)
|
||||||
|
|
||||||
@ -34,11 +34,15 @@ class SDXLTokenizer:
|
|||||||
def untokenize(self, token_weight_pair):
|
def untokenize(self, token_weight_pair):
|
||||||
return self.clip_g.untokenize(token_weight_pair)
|
return self.clip_g.untokenize(token_weight_pair)
|
||||||
|
|
||||||
|
def state_dict(self):
|
||||||
|
return {}
|
||||||
|
|
||||||
class SDXLClipModel(torch.nn.Module):
|
class SDXLClipModel(torch.nn.Module):
|
||||||
def __init__(self, device="cpu", dtype=None):
|
def __init__(self, device="cpu", dtype=None):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.clip_l = sd1_clip.SDClipModel(layer="hidden", layer_idx=-2, device=device, dtype=dtype, layer_norm_hidden_state=False)
|
self.clip_l = sd1_clip.SDClipModel(layer="hidden", layer_idx=-2, device=device, dtype=dtype, layer_norm_hidden_state=False)
|
||||||
self.clip_g = SDXLClipG(device=device, dtype=dtype)
|
self.clip_g = SDXLClipG(device=device, dtype=dtype)
|
||||||
|
self.dtypes = set([dtype])
|
||||||
|
|
||||||
def set_clip_options(self, options):
|
def set_clip_options(self, options):
|
||||||
self.clip_l.set_clip_options(options)
|
self.clip_l.set_clip_options(options)
|
||||||
@ -67,12 +71,12 @@ class SDXLRefinerClipModel(sd1_clip.SD1ClipModel):
|
|||||||
|
|
||||||
|
|
||||||
class StableCascadeClipGTokenizer(sd1_clip.SDTokenizer):
|
class StableCascadeClipGTokenizer(sd1_clip.SDTokenizer):
|
||||||
def __init__(self, tokenizer_path=None, embedding_directory=None):
|
def __init__(self, tokenizer_path=None, embedding_directory=None, tokenizer_data={}):
|
||||||
super().__init__(tokenizer_path, pad_with_end=True, embedding_directory=embedding_directory, embedding_size=1280, embedding_key='clip_g')
|
super().__init__(tokenizer_path, pad_with_end=True, embedding_directory=embedding_directory, embedding_size=1280, embedding_key='clip_g')
|
||||||
|
|
||||||
class StableCascadeTokenizer(sd1_clip.SD1Tokenizer):
|
class StableCascadeTokenizer(sd1_clip.SD1Tokenizer):
|
||||||
def __init__(self, embedding_directory=None):
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
super().__init__(embedding_directory=embedding_directory, clip_name="g", tokenizer=StableCascadeClipGTokenizer)
|
super().__init__(embedding_directory=embedding_directory, tokenizer_data=tokenizer_data, clip_name="g", tokenizer=StableCascadeClipGTokenizer)
|
||||||
|
|
||||||
class StableCascadeClipG(sd1_clip.SDClipModel):
|
class StableCascadeClipG(sd1_clip.SDClipModel):
|
||||||
def __init__(self, device="cpu", max_length=77, freeze=True, layer="hidden", layer_idx=-1, dtype=None):
|
def __init__(self, device="cpu", max_length=77, freeze=True, layer="hidden", layer_idx=-1, dtype=None):
|
||||||
|
|||||||
@ -5,6 +5,10 @@ from . import utils
|
|||||||
from . import sd1_clip
|
from . import sd1_clip
|
||||||
from . import sd2_clip
|
from . import sd2_clip
|
||||||
from . import sdxl_clip
|
from . import sdxl_clip
|
||||||
|
import comfy.text_encoders.sd3_clip
|
||||||
|
import comfy.text_encoders.sa_t5
|
||||||
|
import comfy.text_encoders.aura_t5
|
||||||
|
import comfy.text_encoders.hydit
|
||||||
|
|
||||||
from . import supported_models_base
|
from . import supported_models_base
|
||||||
from . import latent_formats
|
from . import latent_formats
|
||||||
@ -53,7 +57,7 @@ class SD15(supported_models_base.BASE):
|
|||||||
replace_prefix = {"clip_l.": "cond_stage_model."}
|
replace_prefix = {"clip_l.": "cond_stage_model."}
|
||||||
return utils.state_dict_prefix_replace(state_dict, replace_prefix)
|
return utils.state_dict_prefix_replace(state_dict, replace_prefix)
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return supported_models_base.ClipTarget(sd1_clip.SD1Tokenizer, sd1_clip.SD1ClipModel)
|
return supported_models_base.ClipTarget(sd1_clip.SD1Tokenizer, sd1_clip.SD1ClipModel)
|
||||||
|
|
||||||
class SD20(supported_models_base.BASE):
|
class SD20(supported_models_base.BASE):
|
||||||
@ -96,7 +100,7 @@ class SD20(supported_models_base.BASE):
|
|||||||
state_dict = diffusers_convert.convert_text_enc_state_dict_v20(state_dict)
|
state_dict = diffusers_convert.convert_text_enc_state_dict_v20(state_dict)
|
||||||
return state_dict
|
return state_dict
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return supported_models_base.ClipTarget(sd2_clip.SD2Tokenizer, sd2_clip.SD2ClipModel)
|
return supported_models_base.ClipTarget(sd2_clip.SD2Tokenizer, sd2_clip.SD2ClipModel)
|
||||||
|
|
||||||
class SD21UnclipL(SD20):
|
class SD21UnclipL(SD20):
|
||||||
@ -158,7 +162,7 @@ class SDXLRefiner(supported_models_base.BASE):
|
|||||||
state_dict_g = utils.state_dict_prefix_replace(state_dict_g, replace_prefix)
|
state_dict_g = utils.state_dict_prefix_replace(state_dict_g, replace_prefix)
|
||||||
return state_dict_g
|
return state_dict_g
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return supported_models_base.ClipTarget(sdxl_clip.SDXLTokenizer, sdxl_clip.SDXLRefinerClipModel)
|
return supported_models_base.ClipTarget(sdxl_clip.SDXLTokenizer, sdxl_clip.SDXLRefinerClipModel)
|
||||||
|
|
||||||
class SDXL(supported_models_base.BASE):
|
class SDXL(supported_models_base.BASE):
|
||||||
@ -227,7 +231,7 @@ class SDXL(supported_models_base.BASE):
|
|||||||
state_dict_g = utils.state_dict_prefix_replace(state_dict_g, replace_prefix)
|
state_dict_g = utils.state_dict_prefix_replace(state_dict_g, replace_prefix)
|
||||||
return state_dict_g
|
return state_dict_g
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return supported_models_base.ClipTarget(sdxl_clip.SDXLTokenizer, sdxl_clip.SDXLClipModel)
|
return supported_models_base.ClipTarget(sdxl_clip.SDXLTokenizer, sdxl_clip.SDXLClipModel)
|
||||||
|
|
||||||
class SSD1B(SDXL):
|
class SSD1B(SDXL):
|
||||||
@ -298,7 +302,7 @@ class SVD_img2vid(supported_models_base.BASE):
|
|||||||
out = model_base.SVD_img2vid(self, device=device)
|
out = model_base.SVD_img2vid(self, device=device)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return None
|
return None
|
||||||
|
|
||||||
class SV3D_u(SVD_img2vid):
|
class SV3D_u(SVD_img2vid):
|
||||||
@ -364,7 +368,7 @@ class Stable_Zero123(supported_models_base.BASE):
|
|||||||
out = model_base.Stable_Zero123(self, device=device, cc_projection_weight=state_dict["cc_projection.weight"], cc_projection_bias=state_dict["cc_projection.bias"])
|
out = model_base.Stable_Zero123(self, device=device, cc_projection_weight=state_dict["cc_projection.weight"], cc_projection_bias=state_dict["cc_projection.bias"])
|
||||||
return out
|
return out
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return None
|
return None
|
||||||
|
|
||||||
class SD_X4Upscaler(SD20):
|
class SD_X4Upscaler(SD20):
|
||||||
@ -438,7 +442,7 @@ class Stable_Cascade_C(supported_models_base.BASE):
|
|||||||
out = model_base.StableCascade_C(self, device=device)
|
out = model_base.StableCascade_C(self, device=device)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
def clip_target(self):
|
def clip_target(self, state_dict={}):
|
||||||
return supported_models_base.ClipTarget(sdxl_clip.StableCascadeTokenizer, sdxl_clip.StableCascadeClipModel)
|
return supported_models_base.ClipTarget(sdxl_clip.StableCascadeTokenizer, sdxl_clip.StableCascadeClipModel)
|
||||||
|
|
||||||
class Stable_Cascade_B(Stable_Cascade_C):
|
class Stable_Cascade_B(Stable_Cascade_C):
|
||||||
@ -488,6 +492,134 @@ class SDXL_instructpix2pix(SDXL):
|
|||||||
def get_model(self, state_dict, prefix="", device=None):
|
def get_model(self, state_dict, prefix="", device=None):
|
||||||
return model_base.SDXL_instructpix2pix(self, model_type=self.model_type(state_dict, prefix), device=device)
|
return model_base.SDXL_instructpix2pix(self, model_type=self.model_type(state_dict, prefix), device=device)
|
||||||
|
|
||||||
models = [Stable_Zero123, SD15_instructpix2pix, SD15, SD20, SD21UnclipL, SD21UnclipH, SDXL_instructpix2pix, SDXLRefiner, SDXL, SSD1B, KOALA_700M, KOALA_1B, Segmind_Vega, SD_X4Upscaler, Stable_Cascade_C, Stable_Cascade_B, SV3D_u, SV3D_p]
|
class SD3(supported_models_base.BASE):
|
||||||
|
unet_config = {
|
||||||
|
"in_channels": 16,
|
||||||
|
"pos_embed_scaling_factor": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
sampling_settings = {
|
||||||
|
"shift": 3.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
unet_extra_config = {}
|
||||||
|
latent_format = latent_formats.SD3
|
||||||
|
text_encoder_key_prefix = ["text_encoders."]
|
||||||
|
|
||||||
|
def get_model(self, state_dict, prefix="", device=None):
|
||||||
|
out = model_base.SD3(self, device=device)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def clip_target(self, state_dict={}):
|
||||||
|
clip_l = False
|
||||||
|
clip_g = False
|
||||||
|
t5 = False
|
||||||
|
dtype_t5 = None
|
||||||
|
pref = self.text_encoder_key_prefix[0]
|
||||||
|
if "{}clip_l.transformer.text_model.final_layer_norm.weight".format(pref) in state_dict:
|
||||||
|
clip_l = True
|
||||||
|
if "{}clip_g.transformer.text_model.final_layer_norm.weight".format(pref) in state_dict:
|
||||||
|
clip_g = True
|
||||||
|
t5_key = "{}t5xxl.transformer.encoder.final_layer_norm.weight".format(pref)
|
||||||
|
if t5_key in state_dict:
|
||||||
|
t5 = True
|
||||||
|
dtype_t5 = state_dict[t5_key].dtype
|
||||||
|
|
||||||
|
return supported_models_base.ClipTarget(comfy.text_encoders.sd3_clip.SD3Tokenizer, comfy.text_encoders.sd3_clip.sd3_clip(clip_l=clip_l, clip_g=clip_g, t5=t5, dtype_t5=dtype_t5))
|
||||||
|
|
||||||
|
class StableAudio(supported_models_base.BASE):
|
||||||
|
unet_config = {
|
||||||
|
"audio_model": "dit1.0",
|
||||||
|
}
|
||||||
|
|
||||||
|
sampling_settings = {"sigma_max": 500.0, "sigma_min": 0.03}
|
||||||
|
|
||||||
|
unet_extra_config = {}
|
||||||
|
latent_format = latent_formats.StableAudio1
|
||||||
|
|
||||||
|
text_encoder_key_prefix = ["text_encoders."]
|
||||||
|
vae_key_prefix = ["pretransform.model."]
|
||||||
|
|
||||||
|
def get_model(self, state_dict, prefix="", device=None):
|
||||||
|
seconds_start_sd = utils.state_dict_prefix_replace(state_dict, {"conditioner.conditioners.seconds_start.": ""}, filter_keys=True)
|
||||||
|
seconds_total_sd = utils.state_dict_prefix_replace(state_dict, {"conditioner.conditioners.seconds_total.": ""}, filter_keys=True)
|
||||||
|
return model_base.StableAudio1(self, seconds_start_embedder_weights=seconds_start_sd, seconds_total_embedder_weights=seconds_total_sd, device=device)
|
||||||
|
|
||||||
|
def process_unet_state_dict(self, state_dict):
|
||||||
|
for k in list(state_dict.keys()):
|
||||||
|
if k.endswith(".cross_attend_norm.beta") or k.endswith(".ff_norm.beta") or k.endswith(".pre_norm.beta"): #These weights are all zero
|
||||||
|
state_dict.pop(k)
|
||||||
|
return state_dict
|
||||||
|
|
||||||
|
def process_unet_state_dict_for_saving(self, state_dict):
|
||||||
|
replace_prefix = {"": "model.model."}
|
||||||
|
return utils.state_dict_prefix_replace(state_dict, replace_prefix)
|
||||||
|
|
||||||
|
def clip_target(self, state_dict={}):
|
||||||
|
return supported_models_base.ClipTarget(comfy.text_encoders.sa_t5.SAT5Tokenizer, comfy.text_encoders.sa_t5.SAT5Model)
|
||||||
|
|
||||||
|
class AuraFlow(supported_models_base.BASE):
|
||||||
|
unet_config = {
|
||||||
|
"cond_seq_dim": 2048,
|
||||||
|
}
|
||||||
|
|
||||||
|
sampling_settings = {
|
||||||
|
"multiplier": 1.0,
|
||||||
|
"shift": 1.73,
|
||||||
|
}
|
||||||
|
|
||||||
|
unet_extra_config = {}
|
||||||
|
latent_format = latent_formats.SDXL
|
||||||
|
|
||||||
|
vae_key_prefix = ["vae."]
|
||||||
|
text_encoder_key_prefix = ["text_encoders."]
|
||||||
|
|
||||||
|
def get_model(self, state_dict, prefix="", device=None):
|
||||||
|
out = model_base.AuraFlow(self, device=device)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def clip_target(self, state_dict={}):
|
||||||
|
return supported_models_base.ClipTarget(comfy.text_encoders.aura_t5.AuraT5Tokenizer, comfy.text_encoders.aura_t5.AuraT5Model)
|
||||||
|
|
||||||
|
class HunyuanDiT(supported_models_base.BASE):
|
||||||
|
unet_config = {
|
||||||
|
"image_model": "hydit",
|
||||||
|
}
|
||||||
|
|
||||||
|
unet_extra_config = {
|
||||||
|
"attn_precision": torch.float32,
|
||||||
|
}
|
||||||
|
|
||||||
|
sampling_settings = {
|
||||||
|
"linear_start": 0.00085,
|
||||||
|
"linear_end": 0.018,
|
||||||
|
}
|
||||||
|
|
||||||
|
latent_format = latent_formats.SDXL
|
||||||
|
|
||||||
|
vae_key_prefix = ["vae."]
|
||||||
|
text_encoder_key_prefix = ["text_encoders."]
|
||||||
|
|
||||||
|
def get_model(self, state_dict, prefix="", device=None):
|
||||||
|
out = model_base.HunyuanDiT(self, device=device)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def clip_target(self, state_dict={}):
|
||||||
|
return supported_models_base.ClipTarget(comfy.text_encoders.hydit.HyditTokenizer, comfy.text_encoders.hydit.HyditModel)
|
||||||
|
|
||||||
|
class HunyuanDiT1(HunyuanDiT):
|
||||||
|
unet_config = {
|
||||||
|
"image_model": "hydit1",
|
||||||
|
}
|
||||||
|
|
||||||
|
unet_extra_config = {}
|
||||||
|
|
||||||
|
sampling_settings = {
|
||||||
|
"linear_start" : 0.00085,
|
||||||
|
"linear_end" : 0.03,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
models = [Stable_Zero123, SD15_instructpix2pix, SD15, SD20, SD21UnclipL, SD21UnclipH, SDXL_instructpix2pix, SDXLRefiner, SDXL, SSD1B, KOALA_700M, KOALA_1B, Segmind_Vega, SD_X4Upscaler, Stable_Cascade_C, Stable_Cascade_B, SV3D_u, SV3D_p, SD3, StableAudio, AuraFlow, HunyuanDiT, HunyuanDiT1]
|
||||||
|
|
||||||
models += [SVD_img2vid]
|
models += [SVD_img2vid]
|
||||||
|
|||||||
@ -153,7 +153,13 @@ class Adapter(nn.Module):
|
|||||||
features.append(None)
|
features.append(None)
|
||||||
features.append(x)
|
features.append(x)
|
||||||
|
|
||||||
return features
|
features = features[::-1]
|
||||||
|
|
||||||
|
if self.xl:
|
||||||
|
return {"input": features[1:], "middle": features[:1]}
|
||||||
|
else:
|
||||||
|
return {"input": features}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
class LayerNorm(nn.LayerNorm):
|
class LayerNorm(nn.LayerNorm):
|
||||||
@ -290,4 +296,4 @@ class Adapter_light(nn.Module):
|
|||||||
features.append(None)
|
features.append(None)
|
||||||
features.append(x)
|
features.append(x)
|
||||||
|
|
||||||
return features
|
return {"input": features[::-1]}
|
||||||
|
|||||||
@ -25,18 +25,19 @@ class Block(nn.Module):
|
|||||||
def forward(self, x):
|
def forward(self, x):
|
||||||
return self.fuse(self.conv(x) + self.skip(x))
|
return self.fuse(self.conv(x) + self.skip(x))
|
||||||
|
|
||||||
def Encoder():
|
def Encoder(latent_channels=4):
|
||||||
return nn.Sequential(
|
return nn.Sequential(
|
||||||
conv(3, 64), Block(64, 64),
|
conv(3, 64), Block(64, 64),
|
||||||
conv(64, 64, stride=2, bias=False), Block(64, 64), Block(64, 64), Block(64, 64),
|
conv(64, 64, stride=2, bias=False), Block(64, 64), Block(64, 64), Block(64, 64),
|
||||||
conv(64, 64, stride=2, bias=False), Block(64, 64), Block(64, 64), Block(64, 64),
|
conv(64, 64, stride=2, bias=False), Block(64, 64), Block(64, 64), Block(64, 64),
|
||||||
conv(64, 64, stride=2, bias=False), Block(64, 64), Block(64, 64), Block(64, 64),
|
conv(64, 64, stride=2, bias=False), Block(64, 64), Block(64, 64), Block(64, 64),
|
||||||
conv(64, 4),
|
conv(64, latent_channels),
|
||||||
)
|
)
|
||||||
|
|
||||||
def Decoder():
|
|
||||||
|
def Decoder(latent_channels=4):
|
||||||
return nn.Sequential(
|
return nn.Sequential(
|
||||||
Clamp(), conv(4, 64), nn.ReLU(),
|
Clamp(), conv(latent_channels, 64), nn.ReLU(),
|
||||||
Block(64, 64), Block(64, 64), Block(64, 64), nn.Upsample(scale_factor=2), conv(64, 64, bias=False),
|
Block(64, 64), Block(64, 64), Block(64, 64), nn.Upsample(scale_factor=2), conv(64, 64, bias=False),
|
||||||
Block(64, 64), Block(64, 64), Block(64, 64), nn.Upsample(scale_factor=2), conv(64, 64, bias=False),
|
Block(64, 64), Block(64, 64), Block(64, 64), nn.Upsample(scale_factor=2), conv(64, 64, bias=False),
|
||||||
Block(64, 64), Block(64, 64), Block(64, 64), nn.Upsample(scale_factor=2), conv(64, 64, bias=False),
|
Block(64, 64), Block(64, 64), Block(64, 64), nn.Upsample(scale_factor=2), conv(64, 64, bias=False),
|
||||||
@ -47,12 +48,13 @@ class TAESD(nn.Module):
|
|||||||
latent_magnitude = 3
|
latent_magnitude = 3
|
||||||
latent_shift = 0.5
|
latent_shift = 0.5
|
||||||
|
|
||||||
def __init__(self, encoder_path=None, decoder_path=None):
|
def __init__(self, encoder_path=None, decoder_path=None, latent_channels=4):
|
||||||
"""Initialize pretrained TAESD on the given device from the given checkpoints."""
|
"""Initialize pretrained TAESD on the given device from the given checkpoints."""
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.taesd_encoder = Encoder()
|
self.taesd_encoder = Encoder(latent_channels=latent_channels)
|
||||||
self.taesd_decoder = Decoder()
|
self.taesd_decoder = Decoder(latent_channels=latent_channels)
|
||||||
self.vae_scale = torch.nn.Parameter(torch.tensor(1.0))
|
self.vae_scale = torch.nn.Parameter(torch.tensor(1.0))
|
||||||
|
self.vae_shift = torch.nn.Parameter(torch.tensor(0.0))
|
||||||
if encoder_path is not None:
|
if encoder_path is not None:
|
||||||
self.taesd_encoder.load_state_dict(comfy.utils.load_torch_file(encoder_path, safe_load=True))
|
self.taesd_encoder.load_state_dict(comfy.utils.load_torch_file(encoder_path, safe_load=True))
|
||||||
if decoder_path is not None:
|
if decoder_path is not None:
|
||||||
@ -69,9 +71,9 @@ class TAESD(nn.Module):
|
|||||||
return x.sub(TAESD.latent_shift).mul(2 * TAESD.latent_magnitude)
|
return x.sub(TAESD.latent_shift).mul(2 * TAESD.latent_magnitude)
|
||||||
|
|
||||||
def decode(self, x):
|
def decode(self, x):
|
||||||
x_sample = self.taesd_decoder(x * self.vae_scale)
|
x_sample = self.taesd_decoder((x - self.vae_shift) * self.vae_scale)
|
||||||
x_sample = x_sample.sub(0.5).mul(2)
|
x_sample = x_sample.sub(0.5).mul(2)
|
||||||
return x_sample
|
return x_sample
|
||||||
|
|
||||||
def encode(self, x):
|
def encode(self, x):
|
||||||
return self.taesd_encoder(x * 0.5 + 0.5) / self.vae_scale
|
return (self.taesd_encoder(x * 0.5 + 0.5) / self.vae_scale) + self.vae_shift
|
||||||
|
|||||||
22
comfy/text_encoders/aura_t5.py
Normal file
22
comfy/text_encoders/aura_t5.py
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
from comfy import sd1_clip
|
||||||
|
from .spiece_tokenizer import SPieceTokenizer
|
||||||
|
import comfy.text_encoders.t5
|
||||||
|
import os
|
||||||
|
|
||||||
|
class PT5XlModel(sd1_clip.SDClipModel):
|
||||||
|
def __init__(self, device="cpu", layer="last", layer_idx=None, dtype=None):
|
||||||
|
textmodel_json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "t5_pile_config_xl.json")
|
||||||
|
super().__init__(device=device, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"end": 2, "pad": 1}, model_class=comfy.text_encoders.t5.T5, enable_attention_masks=True, zero_out_masked=True)
|
||||||
|
|
||||||
|
class PT5XlTokenizer(sd1_clip.SDTokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
tokenizer_path = os.path.join(os.path.join(os.path.dirname(os.path.realpath(__file__)), "t5_pile_tokenizer"), "tokenizer.model")
|
||||||
|
super().__init__(tokenizer_path, pad_with_end=False, embedding_size=2048, embedding_key='pile_t5xl', tokenizer_class=SPieceTokenizer, has_start_token=False, pad_to_max_length=False, max_length=99999999, min_length=256, pad_token=1)
|
||||||
|
|
||||||
|
class AuraT5Tokenizer(sd1_clip.SD1Tokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
super().__init__(embedding_directory=embedding_directory, tokenizer_data=tokenizer_data, clip_name="pile_t5xl", tokenizer=PT5XlTokenizer)
|
||||||
|
|
||||||
|
class AuraT5Model(sd1_clip.SD1ClipModel):
|
||||||
|
def __init__(self, device="cpu", dtype=None, **kwargs):
|
||||||
|
super().__init__(device=device, dtype=dtype, name="pile_t5xl", clip_model=PT5XlModel, **kwargs)
|
||||||
139
comfy/text_encoders/bert.py
Normal file
139
comfy/text_encoders/bert.py
Normal file
@ -0,0 +1,139 @@
|
|||||||
|
import torch
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention_for_device
|
||||||
|
|
||||||
|
class BertAttention(torch.nn.Module):
|
||||||
|
def __init__(self, embed_dim, heads, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.heads = heads
|
||||||
|
self.query = operations.Linear(embed_dim, embed_dim, bias=True, dtype=dtype, device=device)
|
||||||
|
self.key = operations.Linear(embed_dim, embed_dim, bias=True, dtype=dtype, device=device)
|
||||||
|
self.value = operations.Linear(embed_dim, embed_dim, bias=True, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
|
||||||
|
def forward(self, x, mask=None, optimized_attention=None):
|
||||||
|
q = self.query(x)
|
||||||
|
k = self.key(x)
|
||||||
|
v = self.value(x)
|
||||||
|
|
||||||
|
out = optimized_attention(q, k, v, self.heads, mask)
|
||||||
|
return out
|
||||||
|
|
||||||
|
class BertOutput(torch.nn.Module):
|
||||||
|
def __init__(self, input_dim, output_dim, layer_norm_eps, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.dense = operations.Linear(input_dim, output_dim, dtype=dtype, device=device)
|
||||||
|
self.LayerNorm = operations.LayerNorm(output_dim, eps=layer_norm_eps, dtype=dtype, device=device)
|
||||||
|
# self.dropout = nn.Dropout(0.0)
|
||||||
|
|
||||||
|
def forward(self, x, y):
|
||||||
|
x = self.dense(x)
|
||||||
|
# hidden_states = self.dropout(hidden_states)
|
||||||
|
x = self.LayerNorm(x + y)
|
||||||
|
return x
|
||||||
|
|
||||||
|
class BertAttentionBlock(torch.nn.Module):
|
||||||
|
def __init__(self, embed_dim, heads, layer_norm_eps, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.self = BertAttention(embed_dim, heads, dtype, device, operations)
|
||||||
|
self.output = BertOutput(embed_dim, embed_dim, layer_norm_eps, dtype, device, operations)
|
||||||
|
|
||||||
|
def forward(self, x, mask, optimized_attention):
|
||||||
|
y = self.self(x, mask, optimized_attention)
|
||||||
|
return self.output(y, x)
|
||||||
|
|
||||||
|
class BertIntermediate(torch.nn.Module):
|
||||||
|
def __init__(self, embed_dim, intermediate_dim, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.dense = operations.Linear(embed_dim, intermediate_dim, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = self.dense(x)
|
||||||
|
return torch.nn.functional.gelu(x)
|
||||||
|
|
||||||
|
|
||||||
|
class BertBlock(torch.nn.Module):
|
||||||
|
def __init__(self, embed_dim, intermediate_dim, heads, layer_norm_eps, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.attention = BertAttentionBlock(embed_dim, heads, layer_norm_eps, dtype, device, operations)
|
||||||
|
self.intermediate = BertIntermediate(embed_dim, intermediate_dim, dtype, device, operations)
|
||||||
|
self.output = BertOutput(intermediate_dim, embed_dim, layer_norm_eps, dtype, device, operations)
|
||||||
|
|
||||||
|
def forward(self, x, mask, optimized_attention):
|
||||||
|
x = self.attention(x, mask, optimized_attention)
|
||||||
|
y = self.intermediate(x)
|
||||||
|
return self.output(y, x)
|
||||||
|
|
||||||
|
class BertEncoder(torch.nn.Module):
|
||||||
|
def __init__(self, num_layers, embed_dim, intermediate_dim, heads, layer_norm_eps, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.layer = torch.nn.ModuleList([BertBlock(embed_dim, intermediate_dim, heads, layer_norm_eps, dtype, device, operations) for i in range(num_layers)])
|
||||||
|
|
||||||
|
def forward(self, x, mask=None, intermediate_output=None):
|
||||||
|
optimized_attention = optimized_attention_for_device(x.device, mask=mask is not None, small_input=True)
|
||||||
|
|
||||||
|
if intermediate_output is not None:
|
||||||
|
if intermediate_output < 0:
|
||||||
|
intermediate_output = len(self.layer) + intermediate_output
|
||||||
|
|
||||||
|
intermediate = None
|
||||||
|
for i, l in enumerate(self.layer):
|
||||||
|
x = l(x, mask, optimized_attention)
|
||||||
|
if i == intermediate_output:
|
||||||
|
intermediate = x.clone()
|
||||||
|
return x, intermediate
|
||||||
|
|
||||||
|
class BertEmbeddings(torch.nn.Module):
|
||||||
|
def __init__(self, vocab_size, max_position_embeddings, type_vocab_size, pad_token_id, embed_dim, layer_norm_eps, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.word_embeddings = torch.nn.Embedding(vocab_size, embed_dim, padding_idx=pad_token_id, dtype=dtype, device=device)
|
||||||
|
self.position_embeddings = torch.nn.Embedding(max_position_embeddings, embed_dim, dtype=dtype, device=device)
|
||||||
|
self.token_type_embeddings = torch.nn.Embedding(type_vocab_size, embed_dim, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
self.LayerNorm = operations.LayerNorm(embed_dim, eps=layer_norm_eps, dtype=dtype, device=device)
|
||||||
|
|
||||||
|
def forward(self, input_tokens, token_type_ids=None):
|
||||||
|
x = self.word_embeddings(input_tokens)
|
||||||
|
x += self.position_embeddings.weight[:x.shape[1]]
|
||||||
|
if token_type_ids is not None:
|
||||||
|
x += self.token_type_embeddings(token_type_ids)
|
||||||
|
else:
|
||||||
|
x += self.token_type_embeddings.weight[0]
|
||||||
|
x = self.LayerNorm(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class BertModel_(torch.nn.Module):
|
||||||
|
def __init__(self, config_dict, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
embed_dim = config_dict["hidden_size"]
|
||||||
|
layer_norm_eps = config_dict["layer_norm_eps"]
|
||||||
|
|
||||||
|
self.embeddings = BertEmbeddings(config_dict["vocab_size"], config_dict["max_position_embeddings"], config_dict["type_vocab_size"], config_dict["pad_token_id"], embed_dim, layer_norm_eps, dtype, device, operations)
|
||||||
|
self.encoder = BertEncoder(config_dict["num_hidden_layers"], embed_dim, config_dict["intermediate_size"], config_dict["num_attention_heads"], layer_norm_eps, dtype, device, operations)
|
||||||
|
|
||||||
|
def forward(self, input_tokens, attention_mask=None, intermediate_output=None, final_layer_norm_intermediate=True):
|
||||||
|
x = self.embeddings(input_tokens)
|
||||||
|
mask = None
|
||||||
|
if attention_mask is not None:
|
||||||
|
mask = 1.0 - attention_mask.to(x.dtype).reshape((attention_mask.shape[0], 1, -1, attention_mask.shape[-1])).expand(attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1])
|
||||||
|
mask = mask.masked_fill(mask.to(torch.bool), float("-inf"))
|
||||||
|
|
||||||
|
x, i = self.encoder(x, mask, intermediate_output)
|
||||||
|
return x, i
|
||||||
|
|
||||||
|
|
||||||
|
class BertModel(torch.nn.Module):
|
||||||
|
def __init__(self, config_dict, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.bert = BertModel_(config_dict, dtype, device, operations)
|
||||||
|
self.num_layers = config_dict["num_hidden_layers"]
|
||||||
|
|
||||||
|
def get_input_embeddings(self):
|
||||||
|
return self.bert.embeddings.word_embeddings
|
||||||
|
|
||||||
|
def set_input_embeddings(self, embeddings):
|
||||||
|
self.bert.embeddings.word_embeddings = embeddings
|
||||||
|
|
||||||
|
def forward(self, *args, **kwargs):
|
||||||
|
return self.bert(*args, **kwargs)
|
||||||
79
comfy/text_encoders/hydit.py
Normal file
79
comfy/text_encoders/hydit.py
Normal file
@ -0,0 +1,79 @@
|
|||||||
|
from comfy import sd1_clip
|
||||||
|
from transformers import BertTokenizer
|
||||||
|
from .spiece_tokenizer import SPieceTokenizer
|
||||||
|
from .bert import BertModel
|
||||||
|
import comfy.text_encoders.t5
|
||||||
|
import os
|
||||||
|
import torch
|
||||||
|
|
||||||
|
class HyditBertModel(sd1_clip.SDClipModel):
|
||||||
|
def __init__(self, device="cpu", layer="last", layer_idx=None, dtype=None):
|
||||||
|
textmodel_json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "hydit_clip.json")
|
||||||
|
super().__init__(device=device, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"start": 101, "end": 102, "pad": 0}, model_class=BertModel, enable_attention_masks=True, return_attention_masks=True)
|
||||||
|
|
||||||
|
class HyditBertTokenizer(sd1_clip.SDTokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
tokenizer_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "hydit_clip_tokenizer")
|
||||||
|
super().__init__(tokenizer_path, pad_with_end=False, embedding_size=1024, embedding_key='chinese_roberta', tokenizer_class=BertTokenizer, pad_to_max_length=False, max_length=512, min_length=77)
|
||||||
|
|
||||||
|
|
||||||
|
class MT5XLModel(sd1_clip.SDClipModel):
|
||||||
|
def __init__(self, device="cpu", layer="last", layer_idx=None, dtype=None):
|
||||||
|
textmodel_json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "mt5_config_xl.json")
|
||||||
|
super().__init__(device=device, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"end": 1, "pad": 0}, model_class=comfy.text_encoders.t5.T5, enable_attention_masks=True, return_attention_masks=True)
|
||||||
|
|
||||||
|
class MT5XLTokenizer(sd1_clip.SDTokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
#tokenizer_path = os.path.join(os.path.join(os.path.dirname(os.path.realpath(__file__)), "mt5_tokenizer"), "spiece.model")
|
||||||
|
tokenizer = tokenizer_data.get("spiece_model", None)
|
||||||
|
super().__init__(tokenizer, pad_with_end=False, embedding_size=2048, embedding_key='mt5xl', tokenizer_class=SPieceTokenizer, has_start_token=False, pad_to_max_length=False, max_length=99999999, min_length=256)
|
||||||
|
|
||||||
|
def state_dict(self):
|
||||||
|
return {"spiece_model": self.tokenizer.serialize_model()}
|
||||||
|
|
||||||
|
class HyditTokenizer:
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
mt5_tokenizer_data = tokenizer_data.get("mt5xl.spiece_model", None)
|
||||||
|
self.hydit_clip = HyditBertTokenizer(embedding_directory=embedding_directory)
|
||||||
|
self.mt5xl = MT5XLTokenizer(tokenizer_data={"spiece_model": mt5_tokenizer_data}, embedding_directory=embedding_directory)
|
||||||
|
|
||||||
|
def tokenize_with_weights(self, text:str, return_word_ids=False):
|
||||||
|
out = {}
|
||||||
|
out["hydit_clip"] = self.hydit_clip.tokenize_with_weights(text, return_word_ids)
|
||||||
|
out["mt5xl"] = self.mt5xl.tokenize_with_weights(text, return_word_ids)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def untokenize(self, token_weight_pair):
|
||||||
|
return self.hydit_clip.untokenize(token_weight_pair)
|
||||||
|
|
||||||
|
def state_dict(self):
|
||||||
|
return {"mt5xl.spiece_model": self.mt5xl.state_dict()["spiece_model"]}
|
||||||
|
|
||||||
|
class HyditModel(torch.nn.Module):
|
||||||
|
def __init__(self, device="cpu", dtype=None):
|
||||||
|
super().__init__()
|
||||||
|
self.hydit_clip = HyditBertModel()
|
||||||
|
self.mt5xl = MT5XLModel()
|
||||||
|
|
||||||
|
self.dtypes = set()
|
||||||
|
if dtype is not None:
|
||||||
|
self.dtypes.add(dtype)
|
||||||
|
|
||||||
|
def encode_token_weights(self, token_weight_pairs):
|
||||||
|
hydit_out = self.hydit_clip.encode_token_weights(token_weight_pairs["hydit_clip"])
|
||||||
|
mt5_out = self.mt5xl.encode_token_weights(token_weight_pairs["mt5xl"])
|
||||||
|
return hydit_out[0], hydit_out[1], {"attention_mask": hydit_out[2]["attention_mask"], "conditioning_mt5xl": mt5_out[0], "attention_mask_mt5xl": mt5_out[2]["attention_mask"]}
|
||||||
|
|
||||||
|
def load_sd(self, sd):
|
||||||
|
if "bert.encoder.layer.0.attention.self.query.weight" in sd:
|
||||||
|
return self.hydit_clip.load_sd(sd)
|
||||||
|
else:
|
||||||
|
return self.mt5xl.load_sd(sd)
|
||||||
|
|
||||||
|
def set_clip_options(self, options):
|
||||||
|
self.hydit_clip.set_clip_options(options)
|
||||||
|
self.mt5xl.set_clip_options(options)
|
||||||
|
|
||||||
|
def reset_clip_options(self):
|
||||||
|
self.hydit_clip.reset_clip_options()
|
||||||
|
self.mt5xl.reset_clip_options()
|
||||||
35
comfy/text_encoders/hydit_clip.json
Normal file
35
comfy/text_encoders/hydit_clip.json
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
{
|
||||||
|
"_name_or_path": "hfl/chinese-roberta-wwm-ext-large",
|
||||||
|
"architectures": [
|
||||||
|
"BertModel"
|
||||||
|
],
|
||||||
|
"attention_probs_dropout_prob": 0.1,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"classifier_dropout": null,
|
||||||
|
"directionality": "bidi",
|
||||||
|
"eos_token_id": 2,
|
||||||
|
"hidden_act": "gelu",
|
||||||
|
"hidden_dropout_prob": 0.1,
|
||||||
|
"hidden_size": 1024,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 4096,
|
||||||
|
"layer_norm_eps": 1e-12,
|
||||||
|
"max_position_embeddings": 512,
|
||||||
|
"model_type": "bert",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_hidden_layers": 24,
|
||||||
|
"output_past": true,
|
||||||
|
"pad_token_id": 0,
|
||||||
|
"pooler_fc_size": 768,
|
||||||
|
"pooler_num_attention_heads": 12,
|
||||||
|
"pooler_num_fc_layers": 3,
|
||||||
|
"pooler_size_per_head": 128,
|
||||||
|
"pooler_type": "first_token_transform",
|
||||||
|
"position_embedding_type": "absolute",
|
||||||
|
"torch_dtype": "float32",
|
||||||
|
"transformers_version": "4.22.1",
|
||||||
|
"type_vocab_size": 2,
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 47020
|
||||||
|
}
|
||||||
|
|
||||||
@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"cls_token": "[CLS]",
|
||||||
|
"mask_token": "[MASK]",
|
||||||
|
"pad_token": "[PAD]",
|
||||||
|
"sep_token": "[SEP]",
|
||||||
|
"unk_token": "[UNK]"
|
||||||
|
}
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
{
|
||||||
|
"cls_token": "[CLS]",
|
||||||
|
"do_basic_tokenize": true,
|
||||||
|
"do_lower_case": true,
|
||||||
|
"mask_token": "[MASK]",
|
||||||
|
"name_or_path": "hfl/chinese-roberta-wwm-ext",
|
||||||
|
"never_split": null,
|
||||||
|
"pad_token": "[PAD]",
|
||||||
|
"sep_token": "[SEP]",
|
||||||
|
"special_tokens_map_file": "/home/chenweifeng/.cache/huggingface/hub/models--hfl--chinese-roberta-wwm-ext/snapshots/5c58d0b8ec1d9014354d691c538661bf00bfdb44/special_tokens_map.json",
|
||||||
|
"strip_accents": null,
|
||||||
|
"tokenize_chinese_chars": true,
|
||||||
|
"tokenizer_class": "BertTokenizer",
|
||||||
|
"unk_token": "[UNK]",
|
||||||
|
"model_max_length": 77
|
||||||
|
}
|
||||||
47020
comfy/text_encoders/hydit_clip_tokenizer/vocab.txt
Normal file
47020
comfy/text_encoders/hydit_clip_tokenizer/vocab.txt
Normal file
File diff suppressed because it is too large
Load Diff
22
comfy/text_encoders/mt5_config_xl.json
Normal file
22
comfy/text_encoders/mt5_config_xl.json
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"d_ff": 5120,
|
||||||
|
"d_kv": 64,
|
||||||
|
"d_model": 2048,
|
||||||
|
"decoder_start_token_id": 0,
|
||||||
|
"dropout_rate": 0.1,
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"dense_act_fn": "gelu_pytorch_tanh",
|
||||||
|
"initializer_factor": 1.0,
|
||||||
|
"is_encoder_decoder": true,
|
||||||
|
"is_gated_act": true,
|
||||||
|
"layer_norm_epsilon": 1e-06,
|
||||||
|
"model_type": "mt5",
|
||||||
|
"num_decoder_layers": 24,
|
||||||
|
"num_heads": 32,
|
||||||
|
"num_layers": 24,
|
||||||
|
"output_past": true,
|
||||||
|
"pad_token_id": 0,
|
||||||
|
"relative_attention_num_buckets": 32,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"vocab_size": 250112
|
||||||
|
}
|
||||||
22
comfy/text_encoders/sa_t5.py
Normal file
22
comfy/text_encoders/sa_t5.py
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
from comfy import sd1_clip
|
||||||
|
from transformers import T5TokenizerFast
|
||||||
|
import comfy.text_encoders.t5
|
||||||
|
import os
|
||||||
|
|
||||||
|
class T5BaseModel(sd1_clip.SDClipModel):
|
||||||
|
def __init__(self, device="cpu", layer="last", layer_idx=None, dtype=None):
|
||||||
|
textmodel_json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "t5_config_base.json")
|
||||||
|
super().__init__(device=device, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"end": 1, "pad": 0}, model_class=comfy.text_encoders.t5.T5, enable_attention_masks=True, zero_out_masked=True)
|
||||||
|
|
||||||
|
class T5BaseTokenizer(sd1_clip.SDTokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
tokenizer_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "t5_tokenizer")
|
||||||
|
super().__init__(tokenizer_path, pad_with_end=False, embedding_size=768, embedding_key='t5base', tokenizer_class=T5TokenizerFast, has_start_token=False, pad_to_max_length=False, max_length=99999999, min_length=128)
|
||||||
|
|
||||||
|
class SAT5Tokenizer(sd1_clip.SD1Tokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
super().__init__(embedding_directory=embedding_directory, tokenizer_data=tokenizer_data, clip_name="t5base", tokenizer=T5BaseTokenizer)
|
||||||
|
|
||||||
|
class SAT5Model(sd1_clip.SD1ClipModel):
|
||||||
|
def __init__(self, device="cpu", dtype=None, **kwargs):
|
||||||
|
super().__init__(device=device, dtype=dtype, name="t5base", clip_model=T5BaseModel, **kwargs)
|
||||||
144
comfy/text_encoders/sd3_clip.py
Normal file
144
comfy/text_encoders/sd3_clip.py
Normal file
@ -0,0 +1,144 @@
|
|||||||
|
from comfy import sd1_clip
|
||||||
|
from comfy import sdxl_clip
|
||||||
|
from transformers import T5TokenizerFast
|
||||||
|
import comfy.text_encoders.t5
|
||||||
|
import torch
|
||||||
|
import os
|
||||||
|
import comfy.model_management
|
||||||
|
import logging
|
||||||
|
|
||||||
|
class T5XXLModel(sd1_clip.SDClipModel):
|
||||||
|
def __init__(self, device="cpu", layer="last", layer_idx=None, dtype=None):
|
||||||
|
textmodel_json_config = os.path.join(os.path.dirname(os.path.realpath(__file__)), "t5_config_xxl.json")
|
||||||
|
super().__init__(device=device, layer=layer, layer_idx=layer_idx, textmodel_json_config=textmodel_json_config, dtype=dtype, special_tokens={"end": 1, "pad": 0}, model_class=comfy.text_encoders.t5.T5)
|
||||||
|
|
||||||
|
class T5XXLTokenizer(sd1_clip.SDTokenizer):
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
tokenizer_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "t5_tokenizer")
|
||||||
|
super().__init__(tokenizer_path, pad_with_end=False, embedding_size=4096, embedding_key='t5xxl', tokenizer_class=T5TokenizerFast, has_start_token=False, pad_to_max_length=False, max_length=99999999, min_length=77)
|
||||||
|
|
||||||
|
|
||||||
|
class SD3Tokenizer:
|
||||||
|
def __init__(self, embedding_directory=None, tokenizer_data={}):
|
||||||
|
self.clip_l = sd1_clip.SDTokenizer(embedding_directory=embedding_directory)
|
||||||
|
self.clip_g = sdxl_clip.SDXLClipGTokenizer(embedding_directory=embedding_directory)
|
||||||
|
self.t5xxl = T5XXLTokenizer(embedding_directory=embedding_directory)
|
||||||
|
|
||||||
|
def tokenize_with_weights(self, text:str, return_word_ids=False):
|
||||||
|
out = {}
|
||||||
|
out["g"] = self.clip_g.tokenize_with_weights(text, return_word_ids)
|
||||||
|
out["l"] = self.clip_l.tokenize_with_weights(text, return_word_ids)
|
||||||
|
out["t5xxl"] = self.t5xxl.tokenize_with_weights(text, return_word_ids)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def untokenize(self, token_weight_pair):
|
||||||
|
return self.clip_g.untokenize(token_weight_pair)
|
||||||
|
|
||||||
|
def state_dict(self):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
class SD3ClipModel(torch.nn.Module):
|
||||||
|
def __init__(self, clip_l=True, clip_g=True, t5=True, dtype_t5=None, device="cpu", dtype=None):
|
||||||
|
super().__init__()
|
||||||
|
self.dtypes = set()
|
||||||
|
if clip_l:
|
||||||
|
self.clip_l = sd1_clip.SDClipModel(layer="hidden", layer_idx=-2, device=device, dtype=dtype, layer_norm_hidden_state=False, return_projected_pooled=False)
|
||||||
|
self.dtypes.add(dtype)
|
||||||
|
else:
|
||||||
|
self.clip_l = None
|
||||||
|
|
||||||
|
if clip_g:
|
||||||
|
self.clip_g = sdxl_clip.SDXLClipG(device=device, dtype=dtype)
|
||||||
|
self.dtypes.add(dtype)
|
||||||
|
else:
|
||||||
|
self.clip_g = None
|
||||||
|
|
||||||
|
if t5:
|
||||||
|
if dtype_t5 is None:
|
||||||
|
dtype_t5 = dtype
|
||||||
|
elif comfy.model_management.dtype_size(dtype_t5) > comfy.model_management.dtype_size(dtype):
|
||||||
|
dtype_t5 = dtype
|
||||||
|
|
||||||
|
if not comfy.model_management.supports_cast(device, dtype_t5):
|
||||||
|
dtype_t5 = dtype
|
||||||
|
|
||||||
|
self.t5xxl = T5XXLModel(device=device, dtype=dtype_t5)
|
||||||
|
self.dtypes.add(dtype_t5)
|
||||||
|
else:
|
||||||
|
self.t5xxl = None
|
||||||
|
|
||||||
|
logging.debug("Created SD3 text encoder with: clip_l {}, clip_g {}, t5xxl {}:{}".format(clip_l, clip_g, t5, dtype_t5))
|
||||||
|
|
||||||
|
def set_clip_options(self, options):
|
||||||
|
if self.clip_l is not None:
|
||||||
|
self.clip_l.set_clip_options(options)
|
||||||
|
if self.clip_g is not None:
|
||||||
|
self.clip_g.set_clip_options(options)
|
||||||
|
if self.t5xxl is not None:
|
||||||
|
self.t5xxl.set_clip_options(options)
|
||||||
|
|
||||||
|
def reset_clip_options(self):
|
||||||
|
if self.clip_l is not None:
|
||||||
|
self.clip_l.reset_clip_options()
|
||||||
|
if self.clip_g is not None:
|
||||||
|
self.clip_g.reset_clip_options()
|
||||||
|
if self.t5xxl is not None:
|
||||||
|
self.t5xxl.reset_clip_options()
|
||||||
|
|
||||||
|
def encode_token_weights(self, token_weight_pairs):
|
||||||
|
token_weight_pairs_l = token_weight_pairs["l"]
|
||||||
|
token_weight_pairs_g = token_weight_pairs["g"]
|
||||||
|
token_weight_pars_t5 = token_weight_pairs["t5xxl"]
|
||||||
|
lg_out = None
|
||||||
|
pooled = None
|
||||||
|
out = None
|
||||||
|
|
||||||
|
if len(token_weight_pairs_g) > 0 or len(token_weight_pairs_l) > 0:
|
||||||
|
if self.clip_l is not None:
|
||||||
|
lg_out, l_pooled = self.clip_l.encode_token_weights(token_weight_pairs_l)
|
||||||
|
else:
|
||||||
|
l_pooled = torch.zeros((1, 768), device=comfy.model_management.intermediate_device())
|
||||||
|
|
||||||
|
if self.clip_g is not None:
|
||||||
|
g_out, g_pooled = self.clip_g.encode_token_weights(token_weight_pairs_g)
|
||||||
|
if lg_out is not None:
|
||||||
|
lg_out = torch.cat([lg_out, g_out], dim=-1)
|
||||||
|
else:
|
||||||
|
lg_out = torch.nn.functional.pad(g_out, (768, 0))
|
||||||
|
else:
|
||||||
|
g_out = None
|
||||||
|
g_pooled = torch.zeros((1, 1280), device=comfy.model_management.intermediate_device())
|
||||||
|
|
||||||
|
if lg_out is not None:
|
||||||
|
lg_out = torch.nn.functional.pad(lg_out, (0, 4096 - lg_out.shape[-1]))
|
||||||
|
out = lg_out
|
||||||
|
pooled = torch.cat((l_pooled, g_pooled), dim=-1)
|
||||||
|
|
||||||
|
if self.t5xxl is not None:
|
||||||
|
t5_out, t5_pooled = self.t5xxl.encode_token_weights(token_weight_pars_t5)
|
||||||
|
if lg_out is not None:
|
||||||
|
out = torch.cat([lg_out, t5_out], dim=-2)
|
||||||
|
else:
|
||||||
|
out = t5_out
|
||||||
|
|
||||||
|
if out is None:
|
||||||
|
out = torch.zeros((1, 77, 4096), device=comfy.model_management.intermediate_device())
|
||||||
|
|
||||||
|
if pooled is None:
|
||||||
|
pooled = torch.zeros((1, 768 + 1280), device=comfy.model_management.intermediate_device())
|
||||||
|
|
||||||
|
return out, pooled
|
||||||
|
|
||||||
|
def load_sd(self, sd):
|
||||||
|
if "text_model.encoder.layers.30.mlp.fc1.weight" in sd:
|
||||||
|
return self.clip_g.load_sd(sd)
|
||||||
|
elif "text_model.encoder.layers.1.mlp.fc1.weight" in sd:
|
||||||
|
return self.clip_l.load_sd(sd)
|
||||||
|
else:
|
||||||
|
return self.t5xxl.load_sd(sd)
|
||||||
|
|
||||||
|
def sd3_clip(clip_l=True, clip_g=True, t5=True, dtype_t5=None):
|
||||||
|
class SD3ClipModel_(SD3ClipModel):
|
||||||
|
def __init__(self, device="cpu", dtype=None):
|
||||||
|
super().__init__(clip_l=clip_l, clip_g=clip_g, t5=t5, dtype_t5=dtype_t5, device=device, dtype=dtype)
|
||||||
|
return SD3ClipModel_
|
||||||
32
comfy/text_encoders/spiece_tokenizer.py
Normal file
32
comfy/text_encoders/spiece_tokenizer.py
Normal file
@ -0,0 +1,32 @@
|
|||||||
|
import os
|
||||||
|
import torch
|
||||||
|
|
||||||
|
class SPieceTokenizer:
|
||||||
|
add_eos = True
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def from_pretrained(path):
|
||||||
|
return SPieceTokenizer(path)
|
||||||
|
|
||||||
|
def __init__(self, tokenizer_path):
|
||||||
|
import sentencepiece
|
||||||
|
if torch.is_tensor(tokenizer_path):
|
||||||
|
tokenizer_path = tokenizer_path.numpy().tobytes()
|
||||||
|
|
||||||
|
if isinstance(tokenizer_path, bytes):
|
||||||
|
self.tokenizer = sentencepiece.SentencePieceProcessor(model_proto=tokenizer_path, add_eos=self.add_eos)
|
||||||
|
else:
|
||||||
|
self.tokenizer = sentencepiece.SentencePieceProcessor(model_file=tokenizer_path, add_eos=self.add_eos)
|
||||||
|
|
||||||
|
def get_vocab(self):
|
||||||
|
out = {}
|
||||||
|
for i in range(self.tokenizer.get_piece_size()):
|
||||||
|
out[self.tokenizer.id_to_piece(i)] = i
|
||||||
|
return out
|
||||||
|
|
||||||
|
def __call__(self, string):
|
||||||
|
out = self.tokenizer.encode(string)
|
||||||
|
return {"input_ids": out}
|
||||||
|
|
||||||
|
def serialize_model(self):
|
||||||
|
return torch.ByteTensor(list(self.tokenizer.serialized_model_proto()))
|
||||||
238
comfy/text_encoders/t5.py
Normal file
238
comfy/text_encoders/t5.py
Normal file
@ -0,0 +1,238 @@
|
|||||||
|
import torch
|
||||||
|
import math
|
||||||
|
from comfy.ldm.modules.attention import optimized_attention_for_device
|
||||||
|
|
||||||
|
class T5LayerNorm(torch.nn.Module):
|
||||||
|
def __init__(self, hidden_size, eps=1e-6, dtype=None, device=None, operations=None):
|
||||||
|
super().__init__()
|
||||||
|
self.weight = torch.nn.Parameter(torch.empty(hidden_size, dtype=dtype, device=device))
|
||||||
|
self.variance_epsilon = eps
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
variance = x.pow(2).mean(-1, keepdim=True)
|
||||||
|
x = x * torch.rsqrt(variance + self.variance_epsilon)
|
||||||
|
return self.weight.to(device=x.device, dtype=x.dtype) * x
|
||||||
|
|
||||||
|
activations = {
|
||||||
|
"gelu_pytorch_tanh": lambda a: torch.nn.functional.gelu(a, approximate="tanh"),
|
||||||
|
"relu": torch.nn.functional.relu,
|
||||||
|
}
|
||||||
|
|
||||||
|
class T5DenseActDense(torch.nn.Module):
|
||||||
|
def __init__(self, model_dim, ff_dim, ff_activation, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.wi = operations.Linear(model_dim, ff_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.wo = operations.Linear(ff_dim, model_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
# self.dropout = nn.Dropout(config.dropout_rate)
|
||||||
|
self.act = activations[ff_activation]
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = self.act(self.wi(x))
|
||||||
|
# x = self.dropout(x)
|
||||||
|
x = self.wo(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
class T5DenseGatedActDense(torch.nn.Module):
|
||||||
|
def __init__(self, model_dim, ff_dim, ff_activation, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.wi_0 = operations.Linear(model_dim, ff_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.wi_1 = operations.Linear(model_dim, ff_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.wo = operations.Linear(ff_dim, model_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
# self.dropout = nn.Dropout(config.dropout_rate)
|
||||||
|
self.act = activations[ff_activation]
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
hidden_gelu = self.act(self.wi_0(x))
|
||||||
|
hidden_linear = self.wi_1(x)
|
||||||
|
x = hidden_gelu * hidden_linear
|
||||||
|
# x = self.dropout(x)
|
||||||
|
x = self.wo(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
class T5LayerFF(torch.nn.Module):
|
||||||
|
def __init__(self, model_dim, ff_dim, ff_activation, gated_act, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
if gated_act:
|
||||||
|
self.DenseReluDense = T5DenseGatedActDense(model_dim, ff_dim, ff_activation, dtype, device, operations)
|
||||||
|
else:
|
||||||
|
self.DenseReluDense = T5DenseActDense(model_dim, ff_dim, ff_activation, dtype, device, operations)
|
||||||
|
|
||||||
|
self.layer_norm = T5LayerNorm(model_dim, dtype=dtype, device=device, operations=operations)
|
||||||
|
# self.dropout = nn.Dropout(config.dropout_rate)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
forwarded_states = self.layer_norm(x)
|
||||||
|
forwarded_states = self.DenseReluDense(forwarded_states)
|
||||||
|
# x = x + self.dropout(forwarded_states)
|
||||||
|
x += forwarded_states
|
||||||
|
return x
|
||||||
|
|
||||||
|
class T5Attention(torch.nn.Module):
|
||||||
|
def __init__(self, model_dim, inner_dim, num_heads, relative_attention_bias, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
# Mesh TensorFlow initialization to avoid scaling before softmax
|
||||||
|
self.q = operations.Linear(model_dim, inner_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.k = operations.Linear(model_dim, inner_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.v = operations.Linear(model_dim, inner_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.o = operations.Linear(inner_dim, model_dim, bias=False, dtype=dtype, device=device)
|
||||||
|
self.num_heads = num_heads
|
||||||
|
|
||||||
|
self.relative_attention_bias = None
|
||||||
|
if relative_attention_bias:
|
||||||
|
self.relative_attention_num_buckets = 32
|
||||||
|
self.relative_attention_max_distance = 128
|
||||||
|
self.relative_attention_bias = torch.nn.Embedding(self.relative_attention_num_buckets, self.num_heads, device=device)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _relative_position_bucket(relative_position, bidirectional=True, num_buckets=32, max_distance=128):
|
||||||
|
"""
|
||||||
|
Adapted from Mesh Tensorflow:
|
||||||
|
https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593
|
||||||
|
|
||||||
|
Translate relative position to a bucket number for relative attention. The relative position is defined as
|
||||||
|
memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to
|
||||||
|
position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for
|
||||||
|
small absolute relative_position and larger buckets for larger absolute relative_positions. All relative
|
||||||
|
positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket.
|
||||||
|
This should allow for more graceful generalization to longer sequences than the model has been trained on
|
||||||
|
|
||||||
|
Args:
|
||||||
|
relative_position: an int32 Tensor
|
||||||
|
bidirectional: a boolean - whether the attention is bidirectional
|
||||||
|
num_buckets: an integer
|
||||||
|
max_distance: an integer
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets)
|
||||||
|
"""
|
||||||
|
relative_buckets = 0
|
||||||
|
if bidirectional:
|
||||||
|
num_buckets //= 2
|
||||||
|
relative_buckets += (relative_position > 0).to(torch.long) * num_buckets
|
||||||
|
relative_position = torch.abs(relative_position)
|
||||||
|
else:
|
||||||
|
relative_position = -torch.min(relative_position, torch.zeros_like(relative_position))
|
||||||
|
# now relative_position is in the range [0, inf)
|
||||||
|
|
||||||
|
# half of the buckets are for exact increments in positions
|
||||||
|
max_exact = num_buckets // 2
|
||||||
|
is_small = relative_position < max_exact
|
||||||
|
|
||||||
|
# The other half of the buckets are for logarithmically bigger bins in positions up to max_distance
|
||||||
|
relative_position_if_large = max_exact + (
|
||||||
|
torch.log(relative_position.float() / max_exact)
|
||||||
|
/ math.log(max_distance / max_exact)
|
||||||
|
* (num_buckets - max_exact)
|
||||||
|
).to(torch.long)
|
||||||
|
relative_position_if_large = torch.min(
|
||||||
|
relative_position_if_large, torch.full_like(relative_position_if_large, num_buckets - 1)
|
||||||
|
)
|
||||||
|
|
||||||
|
relative_buckets += torch.where(is_small, relative_position, relative_position_if_large)
|
||||||
|
return relative_buckets
|
||||||
|
|
||||||
|
def compute_bias(self, query_length, key_length, device):
|
||||||
|
"""Compute binned relative position bias"""
|
||||||
|
context_position = torch.arange(query_length, dtype=torch.long, device=device)[:, None]
|
||||||
|
memory_position = torch.arange(key_length, dtype=torch.long, device=device)[None, :]
|
||||||
|
relative_position = memory_position - context_position # shape (query_length, key_length)
|
||||||
|
relative_position_bucket = self._relative_position_bucket(
|
||||||
|
relative_position, # shape (query_length, key_length)
|
||||||
|
bidirectional=True,
|
||||||
|
num_buckets=self.relative_attention_num_buckets,
|
||||||
|
max_distance=self.relative_attention_max_distance,
|
||||||
|
)
|
||||||
|
values = self.relative_attention_bias(relative_position_bucket) # shape (query_length, key_length, num_heads)
|
||||||
|
values = values.permute([2, 0, 1]).unsqueeze(0) # shape (1, num_heads, query_length, key_length)
|
||||||
|
return values
|
||||||
|
|
||||||
|
def forward(self, x, mask=None, past_bias=None, optimized_attention=None):
|
||||||
|
q = self.q(x)
|
||||||
|
k = self.k(x)
|
||||||
|
v = self.v(x)
|
||||||
|
if self.relative_attention_bias is not None:
|
||||||
|
past_bias = self.compute_bias(x.shape[1], x.shape[1], x.device)
|
||||||
|
|
||||||
|
if past_bias is not None:
|
||||||
|
if mask is not None:
|
||||||
|
mask = mask + past_bias
|
||||||
|
else:
|
||||||
|
mask = past_bias
|
||||||
|
|
||||||
|
out = optimized_attention(q, k * ((k.shape[-1] / self.num_heads) ** 0.5), v, self.num_heads, mask)
|
||||||
|
return self.o(out), past_bias
|
||||||
|
|
||||||
|
class T5LayerSelfAttention(torch.nn.Module):
|
||||||
|
def __init__(self, model_dim, inner_dim, ff_dim, num_heads, relative_attention_bias, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.SelfAttention = T5Attention(model_dim, inner_dim, num_heads, relative_attention_bias, dtype, device, operations)
|
||||||
|
self.layer_norm = T5LayerNorm(model_dim, dtype=dtype, device=device, operations=operations)
|
||||||
|
# self.dropout = nn.Dropout(config.dropout_rate)
|
||||||
|
|
||||||
|
def forward(self, x, mask=None, past_bias=None, optimized_attention=None):
|
||||||
|
normed_hidden_states = self.layer_norm(x)
|
||||||
|
output, past_bias = self.SelfAttention(self.layer_norm(x), mask=mask, past_bias=past_bias, optimized_attention=optimized_attention)
|
||||||
|
# x = x + self.dropout(attention_output)
|
||||||
|
x += output
|
||||||
|
return x, past_bias
|
||||||
|
|
||||||
|
class T5Block(torch.nn.Module):
|
||||||
|
def __init__(self, model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.layer = torch.nn.ModuleList()
|
||||||
|
self.layer.append(T5LayerSelfAttention(model_dim, inner_dim, ff_dim, num_heads, relative_attention_bias, dtype, device, operations))
|
||||||
|
self.layer.append(T5LayerFF(model_dim, ff_dim, ff_activation, gated_act, dtype, device, operations))
|
||||||
|
|
||||||
|
def forward(self, x, mask=None, past_bias=None, optimized_attention=None):
|
||||||
|
x, past_bias = self.layer[0](x, mask, past_bias, optimized_attention)
|
||||||
|
x = self.layer[-1](x)
|
||||||
|
return x, past_bias
|
||||||
|
|
||||||
|
class T5Stack(torch.nn.Module):
|
||||||
|
def __init__(self, num_layers, model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self.block = torch.nn.ModuleList(
|
||||||
|
[T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0)), dtype=dtype, device=device, operations=operations) for i in range(num_layers)]
|
||||||
|
)
|
||||||
|
self.final_layer_norm = T5LayerNorm(model_dim, dtype=dtype, device=device, operations=operations)
|
||||||
|
# self.dropout = nn.Dropout(config.dropout_rate)
|
||||||
|
|
||||||
|
def forward(self, x, attention_mask=None, intermediate_output=None, final_layer_norm_intermediate=True):
|
||||||
|
mask = None
|
||||||
|
if attention_mask is not None:
|
||||||
|
mask = 1.0 - attention_mask.to(x.dtype).reshape((attention_mask.shape[0], 1, -1, attention_mask.shape[-1])).expand(attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1])
|
||||||
|
mask = mask.masked_fill(mask.to(torch.bool), float("-inf"))
|
||||||
|
|
||||||
|
intermediate = None
|
||||||
|
optimized_attention = optimized_attention_for_device(x.device, mask=attention_mask is not None, small_input=True)
|
||||||
|
past_bias = None
|
||||||
|
for i, l in enumerate(self.block):
|
||||||
|
x, past_bias = l(x, mask, past_bias, optimized_attention)
|
||||||
|
if i == intermediate_output:
|
||||||
|
intermediate = x.clone()
|
||||||
|
x = self.final_layer_norm(x)
|
||||||
|
if intermediate is not None and final_layer_norm_intermediate:
|
||||||
|
intermediate = self.final_layer_norm(intermediate)
|
||||||
|
return x, intermediate
|
||||||
|
|
||||||
|
class T5(torch.nn.Module):
|
||||||
|
def __init__(self, config_dict, dtype, device, operations):
|
||||||
|
super().__init__()
|
||||||
|
self.num_layers = config_dict["num_layers"]
|
||||||
|
model_dim = config_dict["d_model"]
|
||||||
|
|
||||||
|
self.encoder = T5Stack(self.num_layers, model_dim, model_dim, config_dict["d_ff"], config_dict["dense_act_fn"], config_dict["is_gated_act"], config_dict["num_heads"], config_dict["model_type"] != "umt5", dtype, device, operations)
|
||||||
|
self.dtype = dtype
|
||||||
|
self.shared = torch.nn.Embedding(config_dict["vocab_size"], model_dim, device=device)
|
||||||
|
|
||||||
|
def get_input_embeddings(self):
|
||||||
|
return self.shared
|
||||||
|
|
||||||
|
def set_input_embeddings(self, embeddings):
|
||||||
|
self.shared = embeddings
|
||||||
|
|
||||||
|
def forward(self, input_ids, *args, **kwargs):
|
||||||
|
x = self.shared(input_ids)
|
||||||
|
return self.encoder(x, *args, **kwargs)
|
||||||
22
comfy/text_encoders/t5_config_base.json
Normal file
22
comfy/text_encoders/t5_config_base.json
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"d_ff": 3072,
|
||||||
|
"d_kv": 64,
|
||||||
|
"d_model": 768,
|
||||||
|
"decoder_start_token_id": 0,
|
||||||
|
"dropout_rate": 0.1,
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"dense_act_fn": "relu",
|
||||||
|
"initializer_factor": 1.0,
|
||||||
|
"is_encoder_decoder": true,
|
||||||
|
"is_gated_act": false,
|
||||||
|
"layer_norm_epsilon": 1e-06,
|
||||||
|
"model_type": "t5",
|
||||||
|
"num_decoder_layers": 12,
|
||||||
|
"num_heads": 12,
|
||||||
|
"num_layers": 12,
|
||||||
|
"output_past": true,
|
||||||
|
"pad_token_id": 0,
|
||||||
|
"relative_attention_num_buckets": 32,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"vocab_size": 32128
|
||||||
|
}
|
||||||
22
comfy/text_encoders/t5_config_xxl.json
Normal file
22
comfy/text_encoders/t5_config_xxl.json
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"d_ff": 10240,
|
||||||
|
"d_kv": 64,
|
||||||
|
"d_model": 4096,
|
||||||
|
"decoder_start_token_id": 0,
|
||||||
|
"dropout_rate": 0.1,
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"dense_act_fn": "gelu_pytorch_tanh",
|
||||||
|
"initializer_factor": 1.0,
|
||||||
|
"is_encoder_decoder": true,
|
||||||
|
"is_gated_act": true,
|
||||||
|
"layer_norm_epsilon": 1e-06,
|
||||||
|
"model_type": "t5",
|
||||||
|
"num_decoder_layers": 24,
|
||||||
|
"num_heads": 64,
|
||||||
|
"num_layers": 24,
|
||||||
|
"output_past": true,
|
||||||
|
"pad_token_id": 0,
|
||||||
|
"relative_attention_num_buckets": 32,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"vocab_size": 32128
|
||||||
|
}
|
||||||
22
comfy/text_encoders/t5_pile_config_xl.json
Normal file
22
comfy/text_encoders/t5_pile_config_xl.json
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"d_ff": 5120,
|
||||||
|
"d_kv": 64,
|
||||||
|
"d_model": 2048,
|
||||||
|
"decoder_start_token_id": 0,
|
||||||
|
"dropout_rate": 0.1,
|
||||||
|
"eos_token_id": 2,
|
||||||
|
"dense_act_fn": "gelu_pytorch_tanh",
|
||||||
|
"initializer_factor": 1.0,
|
||||||
|
"is_encoder_decoder": true,
|
||||||
|
"is_gated_act": true,
|
||||||
|
"layer_norm_epsilon": 1e-06,
|
||||||
|
"model_type": "umt5",
|
||||||
|
"num_decoder_layers": 24,
|
||||||
|
"num_heads": 32,
|
||||||
|
"num_layers": 24,
|
||||||
|
"output_past": true,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"relative_attention_num_buckets": 32,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"vocab_size": 32128
|
||||||
|
}
|
||||||
BIN
comfy/text_encoders/t5_pile_tokenizer/tokenizer.model
Normal file
BIN
comfy/text_encoders/t5_pile_tokenizer/tokenizer.model
Normal file
Binary file not shown.
125
comfy/text_encoders/t5_tokenizer/special_tokens_map.json
Normal file
125
comfy/text_encoders/t5_tokenizer/special_tokens_map.json
Normal file
@ -0,0 +1,125 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<extra_id_0>",
|
||||||
|
"<extra_id_1>",
|
||||||
|
"<extra_id_2>",
|
||||||
|
"<extra_id_3>",
|
||||||
|
"<extra_id_4>",
|
||||||
|
"<extra_id_5>",
|
||||||
|
"<extra_id_6>",
|
||||||
|
"<extra_id_7>",
|
||||||
|
"<extra_id_8>",
|
||||||
|
"<extra_id_9>",
|
||||||
|
"<extra_id_10>",
|
||||||
|
"<extra_id_11>",
|
||||||
|
"<extra_id_12>",
|
||||||
|
"<extra_id_13>",
|
||||||
|
"<extra_id_14>",
|
||||||
|
"<extra_id_15>",
|
||||||
|
"<extra_id_16>",
|
||||||
|
"<extra_id_17>",
|
||||||
|
"<extra_id_18>",
|
||||||
|
"<extra_id_19>",
|
||||||
|
"<extra_id_20>",
|
||||||
|
"<extra_id_21>",
|
||||||
|
"<extra_id_22>",
|
||||||
|
"<extra_id_23>",
|
||||||
|
"<extra_id_24>",
|
||||||
|
"<extra_id_25>",
|
||||||
|
"<extra_id_26>",
|
||||||
|
"<extra_id_27>",
|
||||||
|
"<extra_id_28>",
|
||||||
|
"<extra_id_29>",
|
||||||
|
"<extra_id_30>",
|
||||||
|
"<extra_id_31>",
|
||||||
|
"<extra_id_32>",
|
||||||
|
"<extra_id_33>",
|
||||||
|
"<extra_id_34>",
|
||||||
|
"<extra_id_35>",
|
||||||
|
"<extra_id_36>",
|
||||||
|
"<extra_id_37>",
|
||||||
|
"<extra_id_38>",
|
||||||
|
"<extra_id_39>",
|
||||||
|
"<extra_id_40>",
|
||||||
|
"<extra_id_41>",
|
||||||
|
"<extra_id_42>",
|
||||||
|
"<extra_id_43>",
|
||||||
|
"<extra_id_44>",
|
||||||
|
"<extra_id_45>",
|
||||||
|
"<extra_id_46>",
|
||||||
|
"<extra_id_47>",
|
||||||
|
"<extra_id_48>",
|
||||||
|
"<extra_id_49>",
|
||||||
|
"<extra_id_50>",
|
||||||
|
"<extra_id_51>",
|
||||||
|
"<extra_id_52>",
|
||||||
|
"<extra_id_53>",
|
||||||
|
"<extra_id_54>",
|
||||||
|
"<extra_id_55>",
|
||||||
|
"<extra_id_56>",
|
||||||
|
"<extra_id_57>",
|
||||||
|
"<extra_id_58>",
|
||||||
|
"<extra_id_59>",
|
||||||
|
"<extra_id_60>",
|
||||||
|
"<extra_id_61>",
|
||||||
|
"<extra_id_62>",
|
||||||
|
"<extra_id_63>",
|
||||||
|
"<extra_id_64>",
|
||||||
|
"<extra_id_65>",
|
||||||
|
"<extra_id_66>",
|
||||||
|
"<extra_id_67>",
|
||||||
|
"<extra_id_68>",
|
||||||
|
"<extra_id_69>",
|
||||||
|
"<extra_id_70>",
|
||||||
|
"<extra_id_71>",
|
||||||
|
"<extra_id_72>",
|
||||||
|
"<extra_id_73>",
|
||||||
|
"<extra_id_74>",
|
||||||
|
"<extra_id_75>",
|
||||||
|
"<extra_id_76>",
|
||||||
|
"<extra_id_77>",
|
||||||
|
"<extra_id_78>",
|
||||||
|
"<extra_id_79>",
|
||||||
|
"<extra_id_80>",
|
||||||
|
"<extra_id_81>",
|
||||||
|
"<extra_id_82>",
|
||||||
|
"<extra_id_83>",
|
||||||
|
"<extra_id_84>",
|
||||||
|
"<extra_id_85>",
|
||||||
|
"<extra_id_86>",
|
||||||
|
"<extra_id_87>",
|
||||||
|
"<extra_id_88>",
|
||||||
|
"<extra_id_89>",
|
||||||
|
"<extra_id_90>",
|
||||||
|
"<extra_id_91>",
|
||||||
|
"<extra_id_92>",
|
||||||
|
"<extra_id_93>",
|
||||||
|
"<extra_id_94>",
|
||||||
|
"<extra_id_95>",
|
||||||
|
"<extra_id_96>",
|
||||||
|
"<extra_id_97>",
|
||||||
|
"<extra_id_98>",
|
||||||
|
"<extra_id_99>"
|
||||||
|
],
|
||||||
|
"eos_token": {
|
||||||
|
"content": "</s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<pad>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
129428
comfy/text_encoders/t5_tokenizer/tokenizer.json
Normal file
129428
comfy/text_encoders/t5_tokenizer/tokenizer.json
Normal file
File diff suppressed because one or more lines are too long
939
comfy/text_encoders/t5_tokenizer/tokenizer_config.json
Normal file
939
comfy/text_encoders/t5_tokenizer/tokenizer_config.json
Normal file
@ -0,0 +1,939 @@
|
|||||||
|
{
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"0": {
|
||||||
|
"content": "<pad>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"content": "</s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32000": {
|
||||||
|
"content": "<extra_id_99>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32001": {
|
||||||
|
"content": "<extra_id_98>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32002": {
|
||||||
|
"content": "<extra_id_97>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32003": {
|
||||||
|
"content": "<extra_id_96>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32004": {
|
||||||
|
"content": "<extra_id_95>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32005": {
|
||||||
|
"content": "<extra_id_94>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32006": {
|
||||||
|
"content": "<extra_id_93>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32007": {
|
||||||
|
"content": "<extra_id_92>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32008": {
|
||||||
|
"content": "<extra_id_91>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32009": {
|
||||||
|
"content": "<extra_id_90>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32010": {
|
||||||
|
"content": "<extra_id_89>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32011": {
|
||||||
|
"content": "<extra_id_88>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32012": {
|
||||||
|
"content": "<extra_id_87>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32013": {
|
||||||
|
"content": "<extra_id_86>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32014": {
|
||||||
|
"content": "<extra_id_85>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32015": {
|
||||||
|
"content": "<extra_id_84>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32016": {
|
||||||
|
"content": "<extra_id_83>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32017": {
|
||||||
|
"content": "<extra_id_82>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32018": {
|
||||||
|
"content": "<extra_id_81>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32019": {
|
||||||
|
"content": "<extra_id_80>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32020": {
|
||||||
|
"content": "<extra_id_79>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32021": {
|
||||||
|
"content": "<extra_id_78>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32022": {
|
||||||
|
"content": "<extra_id_77>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32023": {
|
||||||
|
"content": "<extra_id_76>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32024": {
|
||||||
|
"content": "<extra_id_75>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32025": {
|
||||||
|
"content": "<extra_id_74>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32026": {
|
||||||
|
"content": "<extra_id_73>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32027": {
|
||||||
|
"content": "<extra_id_72>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32028": {
|
||||||
|
"content": "<extra_id_71>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32029": {
|
||||||
|
"content": "<extra_id_70>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32030": {
|
||||||
|
"content": "<extra_id_69>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32031": {
|
||||||
|
"content": "<extra_id_68>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32032": {
|
||||||
|
"content": "<extra_id_67>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32033": {
|
||||||
|
"content": "<extra_id_66>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32034": {
|
||||||
|
"content": "<extra_id_65>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32035": {
|
||||||
|
"content": "<extra_id_64>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32036": {
|
||||||
|
"content": "<extra_id_63>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32037": {
|
||||||
|
"content": "<extra_id_62>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32038": {
|
||||||
|
"content": "<extra_id_61>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32039": {
|
||||||
|
"content": "<extra_id_60>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32040": {
|
||||||
|
"content": "<extra_id_59>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32041": {
|
||||||
|
"content": "<extra_id_58>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32042": {
|
||||||
|
"content": "<extra_id_57>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32043": {
|
||||||
|
"content": "<extra_id_56>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32044": {
|
||||||
|
"content": "<extra_id_55>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32045": {
|
||||||
|
"content": "<extra_id_54>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32046": {
|
||||||
|
"content": "<extra_id_53>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32047": {
|
||||||
|
"content": "<extra_id_52>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32048": {
|
||||||
|
"content": "<extra_id_51>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32049": {
|
||||||
|
"content": "<extra_id_50>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32050": {
|
||||||
|
"content": "<extra_id_49>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32051": {
|
||||||
|
"content": "<extra_id_48>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32052": {
|
||||||
|
"content": "<extra_id_47>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32053": {
|
||||||
|
"content": "<extra_id_46>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32054": {
|
||||||
|
"content": "<extra_id_45>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32055": {
|
||||||
|
"content": "<extra_id_44>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32056": {
|
||||||
|
"content": "<extra_id_43>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32057": {
|
||||||
|
"content": "<extra_id_42>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32058": {
|
||||||
|
"content": "<extra_id_41>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32059": {
|
||||||
|
"content": "<extra_id_40>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32060": {
|
||||||
|
"content": "<extra_id_39>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32061": {
|
||||||
|
"content": "<extra_id_38>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32062": {
|
||||||
|
"content": "<extra_id_37>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32063": {
|
||||||
|
"content": "<extra_id_36>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32064": {
|
||||||
|
"content": "<extra_id_35>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32065": {
|
||||||
|
"content": "<extra_id_34>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32066": {
|
||||||
|
"content": "<extra_id_33>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32067": {
|
||||||
|
"content": "<extra_id_32>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32068": {
|
||||||
|
"content": "<extra_id_31>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32069": {
|
||||||
|
"content": "<extra_id_30>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32070": {
|
||||||
|
"content": "<extra_id_29>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32071": {
|
||||||
|
"content": "<extra_id_28>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32072": {
|
||||||
|
"content": "<extra_id_27>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32073": {
|
||||||
|
"content": "<extra_id_26>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32074": {
|
||||||
|
"content": "<extra_id_25>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32075": {
|
||||||
|
"content": "<extra_id_24>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32076": {
|
||||||
|
"content": "<extra_id_23>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32077": {
|
||||||
|
"content": "<extra_id_22>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32078": {
|
||||||
|
"content": "<extra_id_21>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32079": {
|
||||||
|
"content": "<extra_id_20>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32080": {
|
||||||
|
"content": "<extra_id_19>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32081": {
|
||||||
|
"content": "<extra_id_18>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32082": {
|
||||||
|
"content": "<extra_id_17>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32083": {
|
||||||
|
"content": "<extra_id_16>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32084": {
|
||||||
|
"content": "<extra_id_15>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32085": {
|
||||||
|
"content": "<extra_id_14>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32086": {
|
||||||
|
"content": "<extra_id_13>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32087": {
|
||||||
|
"content": "<extra_id_12>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32088": {
|
||||||
|
"content": "<extra_id_11>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32089": {
|
||||||
|
"content": "<extra_id_10>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32090": {
|
||||||
|
"content": "<extra_id_9>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32091": {
|
||||||
|
"content": "<extra_id_8>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32092": {
|
||||||
|
"content": "<extra_id_7>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32093": {
|
||||||
|
"content": "<extra_id_6>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32094": {
|
||||||
|
"content": "<extra_id_5>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32095": {
|
||||||
|
"content": "<extra_id_4>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32096": {
|
||||||
|
"content": "<extra_id_3>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32097": {
|
||||||
|
"content": "<extra_id_2>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32098": {
|
||||||
|
"content": "<extra_id_1>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"32099": {
|
||||||
|
"content": "<extra_id_0>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<extra_id_0>",
|
||||||
|
"<extra_id_1>",
|
||||||
|
"<extra_id_2>",
|
||||||
|
"<extra_id_3>",
|
||||||
|
"<extra_id_4>",
|
||||||
|
"<extra_id_5>",
|
||||||
|
"<extra_id_6>",
|
||||||
|
"<extra_id_7>",
|
||||||
|
"<extra_id_8>",
|
||||||
|
"<extra_id_9>",
|
||||||
|
"<extra_id_10>",
|
||||||
|
"<extra_id_11>",
|
||||||
|
"<extra_id_12>",
|
||||||
|
"<extra_id_13>",
|
||||||
|
"<extra_id_14>",
|
||||||
|
"<extra_id_15>",
|
||||||
|
"<extra_id_16>",
|
||||||
|
"<extra_id_17>",
|
||||||
|
"<extra_id_18>",
|
||||||
|
"<extra_id_19>",
|
||||||
|
"<extra_id_20>",
|
||||||
|
"<extra_id_21>",
|
||||||
|
"<extra_id_22>",
|
||||||
|
"<extra_id_23>",
|
||||||
|
"<extra_id_24>",
|
||||||
|
"<extra_id_25>",
|
||||||
|
"<extra_id_26>",
|
||||||
|
"<extra_id_27>",
|
||||||
|
"<extra_id_28>",
|
||||||
|
"<extra_id_29>",
|
||||||
|
"<extra_id_30>",
|
||||||
|
"<extra_id_31>",
|
||||||
|
"<extra_id_32>",
|
||||||
|
"<extra_id_33>",
|
||||||
|
"<extra_id_34>",
|
||||||
|
"<extra_id_35>",
|
||||||
|
"<extra_id_36>",
|
||||||
|
"<extra_id_37>",
|
||||||
|
"<extra_id_38>",
|
||||||
|
"<extra_id_39>",
|
||||||
|
"<extra_id_40>",
|
||||||
|
"<extra_id_41>",
|
||||||
|
"<extra_id_42>",
|
||||||
|
"<extra_id_43>",
|
||||||
|
"<extra_id_44>",
|
||||||
|
"<extra_id_45>",
|
||||||
|
"<extra_id_46>",
|
||||||
|
"<extra_id_47>",
|
||||||
|
"<extra_id_48>",
|
||||||
|
"<extra_id_49>",
|
||||||
|
"<extra_id_50>",
|
||||||
|
"<extra_id_51>",
|
||||||
|
"<extra_id_52>",
|
||||||
|
"<extra_id_53>",
|
||||||
|
"<extra_id_54>",
|
||||||
|
"<extra_id_55>",
|
||||||
|
"<extra_id_56>",
|
||||||
|
"<extra_id_57>",
|
||||||
|
"<extra_id_58>",
|
||||||
|
"<extra_id_59>",
|
||||||
|
"<extra_id_60>",
|
||||||
|
"<extra_id_61>",
|
||||||
|
"<extra_id_62>",
|
||||||
|
"<extra_id_63>",
|
||||||
|
"<extra_id_64>",
|
||||||
|
"<extra_id_65>",
|
||||||
|
"<extra_id_66>",
|
||||||
|
"<extra_id_67>",
|
||||||
|
"<extra_id_68>",
|
||||||
|
"<extra_id_69>",
|
||||||
|
"<extra_id_70>",
|
||||||
|
"<extra_id_71>",
|
||||||
|
"<extra_id_72>",
|
||||||
|
"<extra_id_73>",
|
||||||
|
"<extra_id_74>",
|
||||||
|
"<extra_id_75>",
|
||||||
|
"<extra_id_76>",
|
||||||
|
"<extra_id_77>",
|
||||||
|
"<extra_id_78>",
|
||||||
|
"<extra_id_79>",
|
||||||
|
"<extra_id_80>",
|
||||||
|
"<extra_id_81>",
|
||||||
|
"<extra_id_82>",
|
||||||
|
"<extra_id_83>",
|
||||||
|
"<extra_id_84>",
|
||||||
|
"<extra_id_85>",
|
||||||
|
"<extra_id_86>",
|
||||||
|
"<extra_id_87>",
|
||||||
|
"<extra_id_88>",
|
||||||
|
"<extra_id_89>",
|
||||||
|
"<extra_id_90>",
|
||||||
|
"<extra_id_91>",
|
||||||
|
"<extra_id_92>",
|
||||||
|
"<extra_id_93>",
|
||||||
|
"<extra_id_94>",
|
||||||
|
"<extra_id_95>",
|
||||||
|
"<extra_id_96>",
|
||||||
|
"<extra_id_97>",
|
||||||
|
"<extra_id_98>",
|
||||||
|
"<extra_id_99>"
|
||||||
|
],
|
||||||
|
"clean_up_tokenization_spaces": true,
|
||||||
|
"eos_token": "</s>",
|
||||||
|
"extra_ids": 100,
|
||||||
|
"legacy": false,
|
||||||
|
"model_max_length": 512,
|
||||||
|
"pad_token": "<pad>",
|
||||||
|
"sp_model_kwargs": {},
|
||||||
|
"tokenizer_class": "T5Tokenizer",
|
||||||
|
"unk_token": "<unk>"
|
||||||
|
}
|
||||||
@ -25,7 +25,7 @@ class UnetParams(TypedDict):
|
|||||||
timestep: torch.Tensor
|
timestep: torch.Tensor
|
||||||
c: UnetApplyConds
|
c: UnetApplyConds
|
||||||
# List of [0, 1], [0], [1], ...
|
# List of [0, 1], [0], [1], ...
|
||||||
# 0 means unconditional, 1 means conditional
|
# 0 means conditional, 1 means conditional unconditional
|
||||||
cond_or_uncond: List[int]
|
cond_or_uncond: List[int]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
223
comfy/utils.py
223
comfy/utils.py
@ -6,6 +6,7 @@ import safetensors.torch
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
from PIL import Image
|
from PIL import Image
|
||||||
import logging
|
import logging
|
||||||
|
import itertools
|
||||||
|
|
||||||
def load_torch_file(ckpt, safe_load=False, device=None):
|
def load_torch_file(ckpt, safe_load=False, device=None):
|
||||||
if device is None:
|
if device is None:
|
||||||
@ -249,11 +250,163 @@ def unet_to_diffusers(unet_config):
|
|||||||
|
|
||||||
return diffusers_unet_map
|
return diffusers_unet_map
|
||||||
|
|
||||||
def repeat_to_batch_size(tensor, batch_size):
|
def swap_scale_shift(weight):
|
||||||
if tensor.shape[0] > batch_size:
|
shift, scale = weight.chunk(2, dim=0)
|
||||||
return tensor[:batch_size]
|
new_weight = torch.cat([scale, shift], dim=0)
|
||||||
elif tensor.shape[0] < batch_size:
|
return new_weight
|
||||||
return tensor.repeat([math.ceil(batch_size / tensor.shape[0])] + [1] * (len(tensor.shape) - 1))[:batch_size]
|
|
||||||
|
MMDIT_MAP_BASIC = {
|
||||||
|
("context_embedder.bias", "context_embedder.bias"),
|
||||||
|
("context_embedder.weight", "context_embedder.weight"),
|
||||||
|
("t_embedder.mlp.0.bias", "time_text_embed.timestep_embedder.linear_1.bias"),
|
||||||
|
("t_embedder.mlp.0.weight", "time_text_embed.timestep_embedder.linear_1.weight"),
|
||||||
|
("t_embedder.mlp.2.bias", "time_text_embed.timestep_embedder.linear_2.bias"),
|
||||||
|
("t_embedder.mlp.2.weight", "time_text_embed.timestep_embedder.linear_2.weight"),
|
||||||
|
("x_embedder.proj.bias", "pos_embed.proj.bias"),
|
||||||
|
("x_embedder.proj.weight", "pos_embed.proj.weight"),
|
||||||
|
("y_embedder.mlp.0.bias", "time_text_embed.text_embedder.linear_1.bias"),
|
||||||
|
("y_embedder.mlp.0.weight", "time_text_embed.text_embedder.linear_1.weight"),
|
||||||
|
("y_embedder.mlp.2.bias", "time_text_embed.text_embedder.linear_2.bias"),
|
||||||
|
("y_embedder.mlp.2.weight", "time_text_embed.text_embedder.linear_2.weight"),
|
||||||
|
("pos_embed", "pos_embed.pos_embed"),
|
||||||
|
("final_layer.adaLN_modulation.1.bias", "norm_out.linear.bias", swap_scale_shift),
|
||||||
|
("final_layer.adaLN_modulation.1.weight", "norm_out.linear.weight", swap_scale_shift),
|
||||||
|
("final_layer.linear.bias", "proj_out.bias"),
|
||||||
|
("final_layer.linear.weight", "proj_out.weight"),
|
||||||
|
}
|
||||||
|
|
||||||
|
MMDIT_MAP_BLOCK = {
|
||||||
|
("context_block.adaLN_modulation.1.bias", "norm1_context.linear.bias"),
|
||||||
|
("context_block.adaLN_modulation.1.weight", "norm1_context.linear.weight"),
|
||||||
|
("context_block.attn.proj.bias", "attn.to_add_out.bias"),
|
||||||
|
("context_block.attn.proj.weight", "attn.to_add_out.weight"),
|
||||||
|
("context_block.mlp.fc1.bias", "ff_context.net.0.proj.bias"),
|
||||||
|
("context_block.mlp.fc1.weight", "ff_context.net.0.proj.weight"),
|
||||||
|
("context_block.mlp.fc2.bias", "ff_context.net.2.bias"),
|
||||||
|
("context_block.mlp.fc2.weight", "ff_context.net.2.weight"),
|
||||||
|
("x_block.adaLN_modulation.1.bias", "norm1.linear.bias"),
|
||||||
|
("x_block.adaLN_modulation.1.weight", "norm1.linear.weight"),
|
||||||
|
("x_block.attn.proj.bias", "attn.to_out.0.bias"),
|
||||||
|
("x_block.attn.proj.weight", "attn.to_out.0.weight"),
|
||||||
|
("x_block.mlp.fc1.bias", "ff.net.0.proj.bias"),
|
||||||
|
("x_block.mlp.fc1.weight", "ff.net.0.proj.weight"),
|
||||||
|
("x_block.mlp.fc2.bias", "ff.net.2.bias"),
|
||||||
|
("x_block.mlp.fc2.weight", "ff.net.2.weight"),
|
||||||
|
}
|
||||||
|
|
||||||
|
def mmdit_to_diffusers(mmdit_config, output_prefix=""):
|
||||||
|
key_map = {}
|
||||||
|
|
||||||
|
depth = mmdit_config.get("depth", 0)
|
||||||
|
num_blocks = mmdit_config.get("num_blocks", depth)
|
||||||
|
for i in range(num_blocks):
|
||||||
|
block_from = "transformer_blocks.{}".format(i)
|
||||||
|
block_to = "{}joint_blocks.{}".format(output_prefix, i)
|
||||||
|
|
||||||
|
offset = depth * 64
|
||||||
|
|
||||||
|
for end in ("weight", "bias"):
|
||||||
|
k = "{}.attn.".format(block_from)
|
||||||
|
qkv = "{}.x_block.attn.qkv.{}".format(block_to, end)
|
||||||
|
key_map["{}to_q.{}".format(k, end)] = (qkv, (0, 0, offset))
|
||||||
|
key_map["{}to_k.{}".format(k, end)] = (qkv, (0, offset, offset))
|
||||||
|
key_map["{}to_v.{}".format(k, end)] = (qkv, (0, offset * 2, offset))
|
||||||
|
|
||||||
|
qkv = "{}.context_block.attn.qkv.{}".format(block_to, end)
|
||||||
|
key_map["{}add_q_proj.{}".format(k, end)] = (qkv, (0, 0, offset))
|
||||||
|
key_map["{}add_k_proj.{}".format(k, end)] = (qkv, (0, offset, offset))
|
||||||
|
key_map["{}add_v_proj.{}".format(k, end)] = (qkv, (0, offset * 2, offset))
|
||||||
|
|
||||||
|
for k in MMDIT_MAP_BLOCK:
|
||||||
|
key_map["{}.{}".format(block_from, k[1])] = "{}.{}".format(block_to, k[0])
|
||||||
|
|
||||||
|
map_basic = MMDIT_MAP_BASIC.copy()
|
||||||
|
map_basic.add(("joint_blocks.{}.context_block.adaLN_modulation.1.bias".format(depth - 1), "transformer_blocks.{}.norm1_context.linear.bias".format(depth - 1), swap_scale_shift))
|
||||||
|
map_basic.add(("joint_blocks.{}.context_block.adaLN_modulation.1.weight".format(depth - 1), "transformer_blocks.{}.norm1_context.linear.weight".format(depth - 1), swap_scale_shift))
|
||||||
|
|
||||||
|
for k in map_basic:
|
||||||
|
if len(k) > 2:
|
||||||
|
key_map[k[1]] = ("{}{}".format(output_prefix, k[0]), None, k[2])
|
||||||
|
else:
|
||||||
|
key_map[k[1]] = "{}{}".format(output_prefix, k[0])
|
||||||
|
|
||||||
|
return key_map
|
||||||
|
|
||||||
|
|
||||||
|
def auraflow_to_diffusers(mmdit_config, output_prefix=""):
|
||||||
|
n_double_layers = mmdit_config.get("n_double_layers", 0)
|
||||||
|
n_layers = mmdit_config.get("n_layers", 0)
|
||||||
|
|
||||||
|
key_map = {}
|
||||||
|
for i in range(n_layers):
|
||||||
|
if i < n_double_layers:
|
||||||
|
index = i
|
||||||
|
prefix_from = "joint_transformer_blocks"
|
||||||
|
prefix_to = "{}double_layers".format(output_prefix)
|
||||||
|
block_map = {
|
||||||
|
"attn.to_q.weight": "attn.w2q.weight",
|
||||||
|
"attn.to_k.weight": "attn.w2k.weight",
|
||||||
|
"attn.to_v.weight": "attn.w2v.weight",
|
||||||
|
"attn.to_out.0.weight": "attn.w2o.weight",
|
||||||
|
"attn.add_q_proj.weight": "attn.w1q.weight",
|
||||||
|
"attn.add_k_proj.weight": "attn.w1k.weight",
|
||||||
|
"attn.add_v_proj.weight": "attn.w1v.weight",
|
||||||
|
"attn.to_add_out.weight": "attn.w1o.weight",
|
||||||
|
"ff.linear_1.weight": "mlpX.c_fc1.weight",
|
||||||
|
"ff.linear_2.weight": "mlpX.c_fc2.weight",
|
||||||
|
"ff.out_projection.weight": "mlpX.c_proj.weight",
|
||||||
|
"ff_context.linear_1.weight": "mlpC.c_fc1.weight",
|
||||||
|
"ff_context.linear_2.weight": "mlpC.c_fc2.weight",
|
||||||
|
"ff_context.out_projection.weight": "mlpC.c_proj.weight",
|
||||||
|
"norm1.linear.weight": "modX.1.weight",
|
||||||
|
"norm1_context.linear.weight": "modC.1.weight",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
index = i - n_double_layers
|
||||||
|
prefix_from = "single_transformer_blocks"
|
||||||
|
prefix_to = "{}single_layers".format(output_prefix)
|
||||||
|
|
||||||
|
block_map = {
|
||||||
|
"attn.to_q.weight": "attn.w1q.weight",
|
||||||
|
"attn.to_k.weight": "attn.w1k.weight",
|
||||||
|
"attn.to_v.weight": "attn.w1v.weight",
|
||||||
|
"attn.to_out.0.weight": "attn.w1o.weight",
|
||||||
|
"norm1.linear.weight": "modCX.1.weight",
|
||||||
|
"ff.linear_1.weight": "mlp.c_fc1.weight",
|
||||||
|
"ff.linear_2.weight": "mlp.c_fc2.weight",
|
||||||
|
"ff.out_projection.weight": "mlp.c_proj.weight"
|
||||||
|
}
|
||||||
|
|
||||||
|
for k in block_map:
|
||||||
|
key_map["{}.{}.{}".format(prefix_from, index, k)] = "{}.{}.{}".format(prefix_to, index, block_map[k])
|
||||||
|
|
||||||
|
MAP_BASIC = {
|
||||||
|
("positional_encoding", "pos_embed.pos_embed"),
|
||||||
|
("register_tokens", "register_tokens"),
|
||||||
|
("t_embedder.mlp.0.weight", "time_step_proj.linear_1.weight"),
|
||||||
|
("t_embedder.mlp.0.bias", "time_step_proj.linear_1.bias"),
|
||||||
|
("t_embedder.mlp.2.weight", "time_step_proj.linear_2.weight"),
|
||||||
|
("t_embedder.mlp.2.bias", "time_step_proj.linear_2.bias"),
|
||||||
|
("cond_seq_linear.weight", "context_embedder.weight"),
|
||||||
|
("init_x_linear.weight", "pos_embed.proj.weight"),
|
||||||
|
("init_x_linear.bias", "pos_embed.proj.bias"),
|
||||||
|
("final_linear.weight", "proj_out.weight"),
|
||||||
|
("modF.1.weight", "norm_out.linear.weight", swap_scale_shift),
|
||||||
|
}
|
||||||
|
|
||||||
|
for k in MAP_BASIC:
|
||||||
|
if len(k) > 2:
|
||||||
|
key_map[k[1]] = ("{}{}".format(output_prefix, k[0]), None, k[2])
|
||||||
|
else:
|
||||||
|
key_map[k[1]] = "{}{}".format(output_prefix, k[0])
|
||||||
|
|
||||||
|
return key_map
|
||||||
|
|
||||||
|
def repeat_to_batch_size(tensor, batch_size, dim=0):
|
||||||
|
if tensor.shape[dim] > batch_size:
|
||||||
|
return tensor.narrow(dim, 0, batch_size)
|
||||||
|
elif tensor.shape[dim] < batch_size:
|
||||||
|
return tensor.repeat(dim * [1] + [math.ceil(batch_size / tensor.shape[dim])] + [1] * (len(tensor.shape) - 1 - dim)).narrow(dim, 0, batch_size)
|
||||||
return tensor
|
return tensor
|
||||||
|
|
||||||
def resize_to_batch_size(tensor, batch_size):
|
def resize_to_batch_size(tensor, batch_size):
|
||||||
@ -425,34 +578,52 @@ def get_tiled_scale_steps(width, height, tile_x, tile_y, overlap):
|
|||||||
return math.ceil((height / (tile_y - overlap))) * math.ceil((width / (tile_x - overlap)))
|
return math.ceil((height / (tile_y - overlap))) * math.ceil((width / (tile_x - overlap)))
|
||||||
|
|
||||||
@torch.inference_mode()
|
@torch.inference_mode()
|
||||||
def tiled_scale(samples, function, tile_x=64, tile_y=64, overlap = 8, upscale_amount = 4, out_channels = 3, output_device="cpu", pbar = None):
|
def tiled_scale_multidim(samples, function, tile=(64, 64), overlap = 8, upscale_amount = 4, out_channels = 3, output_device="cpu", pbar = None):
|
||||||
output = torch.empty((samples.shape[0], out_channels, round(samples.shape[2] * upscale_amount), round(samples.shape[3] * upscale_amount)), device=output_device)
|
dims = len(tile)
|
||||||
|
output = torch.empty([samples.shape[0], out_channels] + list(map(lambda a: round(a * upscale_amount), samples.shape[2:])), device=output_device)
|
||||||
|
|
||||||
for b in range(samples.shape[0]):
|
for b in range(samples.shape[0]):
|
||||||
s = samples[b:b+1]
|
s = samples[b:b+1]
|
||||||
out = torch.zeros((s.shape[0], out_channels, round(s.shape[2] * upscale_amount), round(s.shape[3] * upscale_amount)), device=output_device)
|
out = torch.zeros([s.shape[0], out_channels] + list(map(lambda a: round(a * upscale_amount), s.shape[2:])), device=output_device)
|
||||||
out_div = torch.zeros((s.shape[0], out_channels, round(s.shape[2] * upscale_amount), round(s.shape[3] * upscale_amount)), device=output_device)
|
out_div = torch.zeros([s.shape[0], out_channels] + list(map(lambda a: round(a * upscale_amount), s.shape[2:])), device=output_device)
|
||||||
for y in range(0, s.shape[2], tile_y - overlap):
|
|
||||||
for x in range(0, s.shape[3], tile_x - overlap):
|
|
||||||
x = max(0, min(s.shape[-1] - overlap, x))
|
|
||||||
y = max(0, min(s.shape[-2] - overlap, y))
|
|
||||||
s_in = s[:,:,y:y+tile_y,x:x+tile_x]
|
|
||||||
|
|
||||||
ps = function(s_in).to(output_device)
|
for it in itertools.product(*map(lambda a: range(0, a[0], a[1] - overlap), zip(s.shape[2:], tile))):
|
||||||
mask = torch.ones_like(ps)
|
s_in = s
|
||||||
feather = round(overlap * upscale_amount)
|
upscaled = []
|
||||||
for t in range(feather):
|
|
||||||
mask[:,:,t:1+t,:] *= ((1.0/feather) * (t + 1))
|
for d in range(dims):
|
||||||
mask[:,:,mask.shape[2] -1 -t: mask.shape[2]-t,:] *= ((1.0/feather) * (t + 1))
|
pos = max(0, min(s.shape[d + 2] - overlap, it[d]))
|
||||||
mask[:,:,:,t:1+t] *= ((1.0/feather) * (t + 1))
|
l = min(tile[d], s.shape[d + 2] - pos)
|
||||||
mask[:,:,:,mask.shape[3]- 1 - t: mask.shape[3]- t] *= ((1.0/feather) * (t + 1))
|
s_in = s_in.narrow(d + 2, pos, l)
|
||||||
out[:,:,round(y*upscale_amount):round((y+tile_y)*upscale_amount),round(x*upscale_amount):round((x+tile_x)*upscale_amount)] += ps * mask
|
upscaled.append(round(pos * upscale_amount))
|
||||||
out_div[:,:,round(y*upscale_amount):round((y+tile_y)*upscale_amount),round(x*upscale_amount):round((x+tile_x)*upscale_amount)] += mask
|
ps = function(s_in).to(output_device)
|
||||||
if pbar is not None:
|
mask = torch.ones_like(ps)
|
||||||
pbar.update(1)
|
feather = round(overlap * upscale_amount)
|
||||||
|
for t in range(feather):
|
||||||
|
for d in range(2, dims + 2):
|
||||||
|
m = mask.narrow(d, t, 1)
|
||||||
|
m *= ((1.0/feather) * (t + 1))
|
||||||
|
m = mask.narrow(d, mask.shape[d] -1 -t, 1)
|
||||||
|
m *= ((1.0/feather) * (t + 1))
|
||||||
|
|
||||||
|
o = out
|
||||||
|
o_d = out_div
|
||||||
|
for d in range(dims):
|
||||||
|
o = o.narrow(d + 2, upscaled[d], mask.shape[d + 2])
|
||||||
|
o_d = o_d.narrow(d + 2, upscaled[d], mask.shape[d + 2])
|
||||||
|
|
||||||
|
o += ps * mask
|
||||||
|
o_d += mask
|
||||||
|
|
||||||
|
if pbar is not None:
|
||||||
|
pbar.update(1)
|
||||||
|
|
||||||
output[b:b+1] = out/out_div
|
output[b:b+1] = out/out_div
|
||||||
return output
|
return output
|
||||||
|
|
||||||
|
def tiled_scale(samples, function, tile_x=64, tile_y=64, overlap = 8, upscale_amount = 4, out_channels = 3, output_device="cpu", pbar = None):
|
||||||
|
return tiled_scale_multidim(samples, function, (tile_y, tile_x), overlap, upscale_amount, out_channels, output_device, pbar)
|
||||||
|
|
||||||
PROGRESS_BAR_ENABLED = True
|
PROGRESS_BAR_ENABLED = True
|
||||||
def set_progress_bar_enabled(enabled):
|
def set_progress_bar_enabled(enabled):
|
||||||
global PROGRESS_BAR_ENABLED
|
global PROGRESS_BAR_ENABLED
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright [yyyy] [name of copyright owner]
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright [yyyy] [name of copyright owner]
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
MIT License
|
|
||||||
|
|
||||||
Copyright (c) 2022 Xiangyu Chen
|
|
||||||
|
|
||||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
||||||
of this software and associated documentation files (the "Software"), to deal
|
|
||||||
in the Software without restriction, including without limitation the rights
|
|
||||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
copies of the Software, and to permit persons to whom the Software is
|
|
||||||
furnished to do so, subject to the following conditions:
|
|
||||||
|
|
||||||
The above copyright notice and this permission notice shall be included in all
|
|
||||||
copies or substantial portions of the Software.
|
|
||||||
|
|
||||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
SOFTWARE.
|
|
||||||
@ -1,29 +0,0 @@
|
|||||||
BSD 3-Clause License
|
|
||||||
|
|
||||||
Copyright (c) 2021, Xintao Wang
|
|
||||||
All rights reserved.
|
|
||||||
|
|
||||||
Redistribution and use in source and binary forms, with or without
|
|
||||||
modification, are permitted provided that the following conditions are met:
|
|
||||||
|
|
||||||
1. Redistributions of source code must retain the above copyright notice, this
|
|
||||||
list of conditions and the following disclaimer.
|
|
||||||
|
|
||||||
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
||||||
this list of conditions and the following disclaimer in the documentation
|
|
||||||
and/or other materials provided with the distribution.
|
|
||||||
|
|
||||||
3. Neither the name of the copyright holder nor the names of its
|
|
||||||
contributors may be used to endorse or promote products derived from
|
|
||||||
this software without specific prior written permission.
|
|
||||||
|
|
||||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
||||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
||||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
||||||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
||||||
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
||||||
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
||||||
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
||||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
||||||
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
||||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright 2022 Kai Zhang (cskaizhang@gmail.com, https://cszn.github.io/). All rights reserved.
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright 2018-2022 BasicSR Authors
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,121 +0,0 @@
|
|||||||
Creative Commons Legal Code
|
|
||||||
|
|
||||||
CC0 1.0 Universal
|
|
||||||
|
|
||||||
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
|
|
||||||
LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
|
|
||||||
ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
|
|
||||||
INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
|
|
||||||
REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
|
|
||||||
PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
|
|
||||||
THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
|
|
||||||
HEREUNDER.
|
|
||||||
|
|
||||||
Statement of Purpose
|
|
||||||
|
|
||||||
The laws of most jurisdictions throughout the world automatically confer
|
|
||||||
exclusive Copyright and Related Rights (defined below) upon the creator
|
|
||||||
and subsequent owner(s) (each and all, an "owner") of an original work of
|
|
||||||
authorship and/or a database (each, a "Work").
|
|
||||||
|
|
||||||
Certain owners wish to permanently relinquish those rights to a Work for
|
|
||||||
the purpose of contributing to a commons of creative, cultural and
|
|
||||||
scientific works ("Commons") that the public can reliably and without fear
|
|
||||||
of later claims of infringement build upon, modify, incorporate in other
|
|
||||||
works, reuse and redistribute as freely as possible in any form whatsoever
|
|
||||||
and for any purposes, including without limitation commercial purposes.
|
|
||||||
These owners may contribute to the Commons to promote the ideal of a free
|
|
||||||
culture and the further production of creative, cultural and scientific
|
|
||||||
works, or to gain reputation or greater distribution for their Work in
|
|
||||||
part through the use and efforts of others.
|
|
||||||
|
|
||||||
For these and/or other purposes and motivations, and without any
|
|
||||||
expectation of additional consideration or compensation, the person
|
|
||||||
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
|
|
||||||
is an owner of Copyright and Related Rights in the Work, voluntarily
|
|
||||||
elects to apply CC0 to the Work and publicly distribute the Work under its
|
|
||||||
terms, with knowledge of his or her Copyright and Related Rights in the
|
|
||||||
Work and the meaning and intended legal effect of CC0 on those rights.
|
|
||||||
|
|
||||||
1. Copyright and Related Rights. A Work made available under CC0 may be
|
|
||||||
protected by copyright and related or neighboring rights ("Copyright and
|
|
||||||
Related Rights"). Copyright and Related Rights include, but are not
|
|
||||||
limited to, the following:
|
|
||||||
|
|
||||||
i. the right to reproduce, adapt, distribute, perform, display,
|
|
||||||
communicate, and translate a Work;
|
|
||||||
ii. moral rights retained by the original author(s) and/or performer(s);
|
|
||||||
iii. publicity and privacy rights pertaining to a person's image or
|
|
||||||
likeness depicted in a Work;
|
|
||||||
iv. rights protecting against unfair competition in regards to a Work,
|
|
||||||
subject to the limitations in paragraph 4(a), below;
|
|
||||||
v. rights protecting the extraction, dissemination, use and reuse of data
|
|
||||||
in a Work;
|
|
||||||
vi. database rights (such as those arising under Directive 96/9/EC of the
|
|
||||||
European Parliament and of the Council of 11 March 1996 on the legal
|
|
||||||
protection of databases, and under any national implementation
|
|
||||||
thereof, including any amended or successor version of such
|
|
||||||
directive); and
|
|
||||||
vii. other similar, equivalent or corresponding rights throughout the
|
|
||||||
world based on applicable law or treaty, and any national
|
|
||||||
implementations thereof.
|
|
||||||
|
|
||||||
2. Waiver. To the greatest extent permitted by, but not in contravention
|
|
||||||
of, applicable law, Affirmer hereby overtly, fully, permanently,
|
|
||||||
irrevocably and unconditionally waives, abandons, and surrenders all of
|
|
||||||
Affirmer's Copyright and Related Rights and associated claims and causes
|
|
||||||
of action, whether now known or unknown (including existing as well as
|
|
||||||
future claims and causes of action), in the Work (i) in all territories
|
|
||||||
worldwide, (ii) for the maximum duration provided by applicable law or
|
|
||||||
treaty (including future time extensions), (iii) in any current or future
|
|
||||||
medium and for any number of copies, and (iv) for any purpose whatsoever,
|
|
||||||
including without limitation commercial, advertising or promotional
|
|
||||||
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
|
|
||||||
member of the public at large and to the detriment of Affirmer's heirs and
|
|
||||||
successors, fully intending that such Waiver shall not be subject to
|
|
||||||
revocation, rescission, cancellation, termination, or any other legal or
|
|
||||||
equitable action to disrupt the quiet enjoyment of the Work by the public
|
|
||||||
as contemplated by Affirmer's express Statement of Purpose.
|
|
||||||
|
|
||||||
3. Public License Fallback. Should any part of the Waiver for any reason
|
|
||||||
be judged legally invalid or ineffective under applicable law, then the
|
|
||||||
Waiver shall be preserved to the maximum extent permitted taking into
|
|
||||||
account Affirmer's express Statement of Purpose. In addition, to the
|
|
||||||
extent the Waiver is so judged Affirmer hereby grants to each affected
|
|
||||||
person a royalty-free, non transferable, non sublicensable, non exclusive,
|
|
||||||
irrevocable and unconditional license to exercise Affirmer's Copyright and
|
|
||||||
Related Rights in the Work (i) in all territories worldwide, (ii) for the
|
|
||||||
maximum duration provided by applicable law or treaty (including future
|
|
||||||
time extensions), (iii) in any current or future medium and for any number
|
|
||||||
of copies, and (iv) for any purpose whatsoever, including without
|
|
||||||
limitation commercial, advertising or promotional purposes (the
|
|
||||||
"License"). The License shall be deemed effective as of the date CC0 was
|
|
||||||
applied by Affirmer to the Work. Should any part of the License for any
|
|
||||||
reason be judged legally invalid or ineffective under applicable law, such
|
|
||||||
partial invalidity or ineffectiveness shall not invalidate the remainder
|
|
||||||
of the License, and in such case Affirmer hereby affirms that he or she
|
|
||||||
will not (i) exercise any of his or her remaining Copyright and Related
|
|
||||||
Rights in the Work or (ii) assert any associated claims and causes of
|
|
||||||
action with respect to the Work, in either case contrary to Affirmer's
|
|
||||||
express Statement of Purpose.
|
|
||||||
|
|
||||||
4. Limitations and Disclaimers.
|
|
||||||
|
|
||||||
a. No trademark or patent rights held by Affirmer are waived, abandoned,
|
|
||||||
surrendered, licensed or otherwise affected by this document.
|
|
||||||
b. Affirmer offers the Work as-is and makes no representations or
|
|
||||||
warranties of any kind concerning the Work, express, implied,
|
|
||||||
statutory or otherwise, including without limitation warranties of
|
|
||||||
title, merchantability, fitness for a particular purpose, non
|
|
||||||
infringement, or the absence of latent or other defects, accuracy, or
|
|
||||||
the present or absence of errors, whether or not discoverable, all to
|
|
||||||
the greatest extent permissible under applicable law.
|
|
||||||
c. Affirmer disclaims responsibility for clearing rights of other persons
|
|
||||||
that may apply to the Work or any use thereof, including without
|
|
||||||
limitation any person's Copyright and Related Rights in the Work.
|
|
||||||
Further, Affirmer disclaims responsibility for obtaining any necessary
|
|
||||||
consents, permissions or other rights required for any use of the
|
|
||||||
Work.
|
|
||||||
d. Affirmer understands and acknowledges that Creative Commons is not a
|
|
||||||
party to this document and has no duty or obligation with respect to
|
|
||||||
this CC0 or use of the Work.
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright [2021] [SwinIR Authors]
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright [2021] [SwinIR Authors]
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright [2021] Samsung Research
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,694 +0,0 @@
|
|||||||
# pylint: skip-file
|
|
||||||
"""
|
|
||||||
Model adapted from advimman's lama project: https://github.com/advimman/lama
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Fast Fourier Convolution NeurIPS 2020
|
|
||||||
# original implementation https://github.com/pkumivision/FFC/blob/main/model_zoo/ffc.py
|
|
||||||
# paper https://proceedings.neurips.cc/paper/2020/file/2fd5d41ec6cfab47e32164d5624269b1-Paper.pdf
|
|
||||||
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
import torch
|
|
||||||
import torch.nn as nn
|
|
||||||
import torch.nn.functional as F
|
|
||||||
from torchvision.transforms.functional import InterpolationMode, rotate
|
|
||||||
|
|
||||||
|
|
||||||
class LearnableSpatialTransformWrapper(nn.Module):
|
|
||||||
def __init__(self, impl, pad_coef=0.5, angle_init_range=80, train_angle=True):
|
|
||||||
super().__init__()
|
|
||||||
self.impl = impl
|
|
||||||
self.angle = torch.rand(1) * angle_init_range
|
|
||||||
if train_angle:
|
|
||||||
self.angle = nn.Parameter(self.angle, requires_grad=True)
|
|
||||||
self.pad_coef = pad_coef
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
if torch.is_tensor(x):
|
|
||||||
return self.inverse_transform(self.impl(self.transform(x)), x)
|
|
||||||
elif isinstance(x, tuple):
|
|
||||||
x_trans = tuple(self.transform(elem) for elem in x)
|
|
||||||
y_trans = self.impl(x_trans)
|
|
||||||
return tuple(
|
|
||||||
self.inverse_transform(elem, orig_x) for elem, orig_x in zip(y_trans, x)
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
raise ValueError(f"Unexpected input type {type(x)}")
|
|
||||||
|
|
||||||
def transform(self, x):
|
|
||||||
height, width = x.shape[2:]
|
|
||||||
pad_h, pad_w = int(height * self.pad_coef), int(width * self.pad_coef)
|
|
||||||
x_padded = F.pad(x, [pad_w, pad_w, pad_h, pad_h], mode="reflect")
|
|
||||||
x_padded_rotated = rotate(
|
|
||||||
x_padded, self.angle.to(x_padded), InterpolationMode.BILINEAR, fill=0
|
|
||||||
)
|
|
||||||
|
|
||||||
return x_padded_rotated
|
|
||||||
|
|
||||||
def inverse_transform(self, y_padded_rotated, orig_x):
|
|
||||||
height, width = orig_x.shape[2:]
|
|
||||||
pad_h, pad_w = int(height * self.pad_coef), int(width * self.pad_coef)
|
|
||||||
|
|
||||||
y_padded = rotate(
|
|
||||||
y_padded_rotated,
|
|
||||||
-self.angle.to(y_padded_rotated),
|
|
||||||
InterpolationMode.BILINEAR,
|
|
||||||
fill=0,
|
|
||||||
)
|
|
||||||
y_height, y_width = y_padded.shape[2:]
|
|
||||||
y = y_padded[:, :, pad_h : y_height - pad_h, pad_w : y_width - pad_w]
|
|
||||||
return y
|
|
||||||
|
|
||||||
|
|
||||||
class SELayer(nn.Module):
|
|
||||||
def __init__(self, channel, reduction=16):
|
|
||||||
super(SELayer, self).__init__()
|
|
||||||
self.avg_pool = nn.AdaptiveAvgPool2d(1)
|
|
||||||
self.fc = nn.Sequential(
|
|
||||||
nn.Linear(channel, channel // reduction, bias=False),
|
|
||||||
nn.ReLU(inplace=True),
|
|
||||||
nn.Linear(channel // reduction, channel, bias=False),
|
|
||||||
nn.Sigmoid(),
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
b, c, _, _ = x.size()
|
|
||||||
y = self.avg_pool(x).view(b, c)
|
|
||||||
y = self.fc(y).view(b, c, 1, 1)
|
|
||||||
res = x * y.expand_as(x)
|
|
||||||
return res
|
|
||||||
|
|
||||||
|
|
||||||
class FourierUnit(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
in_channels,
|
|
||||||
out_channels,
|
|
||||||
groups=1,
|
|
||||||
spatial_scale_factor=None,
|
|
||||||
spatial_scale_mode="bilinear",
|
|
||||||
spectral_pos_encoding=False,
|
|
||||||
use_se=False,
|
|
||||||
se_kwargs=None,
|
|
||||||
ffc3d=False,
|
|
||||||
fft_norm="ortho",
|
|
||||||
):
|
|
||||||
# bn_layer not used
|
|
||||||
super(FourierUnit, self).__init__()
|
|
||||||
self.groups = groups
|
|
||||||
|
|
||||||
self.conv_layer = torch.nn.Conv2d(
|
|
||||||
in_channels=in_channels * 2 + (2 if spectral_pos_encoding else 0),
|
|
||||||
out_channels=out_channels * 2,
|
|
||||||
kernel_size=1,
|
|
||||||
stride=1,
|
|
||||||
padding=0,
|
|
||||||
groups=self.groups,
|
|
||||||
bias=False,
|
|
||||||
)
|
|
||||||
self.bn = torch.nn.BatchNorm2d(out_channels * 2)
|
|
||||||
self.relu = torch.nn.ReLU(inplace=True)
|
|
||||||
|
|
||||||
# squeeze and excitation block
|
|
||||||
self.use_se = use_se
|
|
||||||
if use_se:
|
|
||||||
if se_kwargs is None:
|
|
||||||
se_kwargs = {}
|
|
||||||
self.se = SELayer(self.conv_layer.in_channels, **se_kwargs)
|
|
||||||
|
|
||||||
self.spatial_scale_factor = spatial_scale_factor
|
|
||||||
self.spatial_scale_mode = spatial_scale_mode
|
|
||||||
self.spectral_pos_encoding = spectral_pos_encoding
|
|
||||||
self.ffc3d = ffc3d
|
|
||||||
self.fft_norm = fft_norm
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
half_check = False
|
|
||||||
if x.type() == "torch.cuda.HalfTensor":
|
|
||||||
# half only works on gpu anyway
|
|
||||||
half_check = True
|
|
||||||
|
|
||||||
batch = x.shape[0]
|
|
||||||
|
|
||||||
if self.spatial_scale_factor is not None:
|
|
||||||
orig_size = x.shape[-2:]
|
|
||||||
x = F.interpolate(
|
|
||||||
x,
|
|
||||||
scale_factor=self.spatial_scale_factor,
|
|
||||||
mode=self.spatial_scale_mode,
|
|
||||||
align_corners=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
# (batch, c, h, w/2+1, 2)
|
|
||||||
fft_dim = (-3, -2, -1) if self.ffc3d else (-2, -1)
|
|
||||||
if half_check == True:
|
|
||||||
ffted = torch.fft.rfftn(
|
|
||||||
x.float(), dim=fft_dim, norm=self.fft_norm
|
|
||||||
) # .type(torch.cuda.HalfTensor)
|
|
||||||
else:
|
|
||||||
ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm)
|
|
||||||
|
|
||||||
ffted = torch.stack((ffted.real, ffted.imag), dim=-1)
|
|
||||||
ffted = ffted.permute(0, 1, 4, 2, 3).contiguous() # (batch, c, 2, h, w/2+1)
|
|
||||||
ffted = ffted.view(
|
|
||||||
(
|
|
||||||
batch,
|
|
||||||
-1,
|
|
||||||
)
|
|
||||||
+ ffted.size()[3:]
|
|
||||||
)
|
|
||||||
|
|
||||||
if self.spectral_pos_encoding:
|
|
||||||
height, width = ffted.shape[-2:]
|
|
||||||
coords_vert = (
|
|
||||||
torch.linspace(0, 1, height)[None, None, :, None]
|
|
||||||
.expand(batch, 1, height, width)
|
|
||||||
.to(ffted)
|
|
||||||
)
|
|
||||||
coords_hor = (
|
|
||||||
torch.linspace(0, 1, width)[None, None, None, :]
|
|
||||||
.expand(batch, 1, height, width)
|
|
||||||
.to(ffted)
|
|
||||||
)
|
|
||||||
ffted = torch.cat((coords_vert, coords_hor, ffted), dim=1)
|
|
||||||
|
|
||||||
if self.use_se:
|
|
||||||
ffted = self.se(ffted)
|
|
||||||
|
|
||||||
if half_check == True:
|
|
||||||
ffted = self.conv_layer(ffted.half()) # (batch, c*2, h, w/2+1)
|
|
||||||
else:
|
|
||||||
ffted = self.conv_layer(
|
|
||||||
ffted
|
|
||||||
) # .type(torch.cuda.FloatTensor) # (batch, c*2, h, w/2+1)
|
|
||||||
|
|
||||||
ffted = self.relu(self.bn(ffted))
|
|
||||||
# forcing to be always float
|
|
||||||
ffted = ffted.float()
|
|
||||||
|
|
||||||
ffted = (
|
|
||||||
ffted.view(
|
|
||||||
(
|
|
||||||
batch,
|
|
||||||
-1,
|
|
||||||
2,
|
|
||||||
)
|
|
||||||
+ ffted.size()[2:]
|
|
||||||
)
|
|
||||||
.permute(0, 1, 3, 4, 2)
|
|
||||||
.contiguous()
|
|
||||||
) # (batch,c, t, h, w/2+1, 2)
|
|
||||||
|
|
||||||
ffted = torch.complex(ffted[..., 0], ffted[..., 1])
|
|
||||||
|
|
||||||
ifft_shape_slice = x.shape[-3:] if self.ffc3d else x.shape[-2:]
|
|
||||||
output = torch.fft.irfftn(
|
|
||||||
ffted, s=ifft_shape_slice, dim=fft_dim, norm=self.fft_norm
|
|
||||||
)
|
|
||||||
|
|
||||||
if half_check == True:
|
|
||||||
output = output.half()
|
|
||||||
|
|
||||||
if self.spatial_scale_factor is not None:
|
|
||||||
output = F.interpolate(
|
|
||||||
output,
|
|
||||||
size=orig_size,
|
|
||||||
mode=self.spatial_scale_mode,
|
|
||||||
align_corners=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
return output
|
|
||||||
|
|
||||||
|
|
||||||
class SpectralTransform(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
in_channels,
|
|
||||||
out_channels,
|
|
||||||
stride=1,
|
|
||||||
groups=1,
|
|
||||||
enable_lfu=True,
|
|
||||||
separable_fu=False,
|
|
||||||
**fu_kwargs,
|
|
||||||
):
|
|
||||||
# bn_layer not used
|
|
||||||
super(SpectralTransform, self).__init__()
|
|
||||||
self.enable_lfu = enable_lfu
|
|
||||||
if stride == 2:
|
|
||||||
self.downsample = nn.AvgPool2d(kernel_size=(2, 2), stride=2)
|
|
||||||
else:
|
|
||||||
self.downsample = nn.Identity()
|
|
||||||
|
|
||||||
self.stride = stride
|
|
||||||
self.conv1 = nn.Sequential(
|
|
||||||
nn.Conv2d(
|
|
||||||
in_channels, out_channels // 2, kernel_size=1, groups=groups, bias=False
|
|
||||||
),
|
|
||||||
nn.BatchNorm2d(out_channels // 2),
|
|
||||||
nn.ReLU(inplace=True),
|
|
||||||
)
|
|
||||||
fu_class = FourierUnit
|
|
||||||
self.fu = fu_class(out_channels // 2, out_channels // 2, groups, **fu_kwargs)
|
|
||||||
if self.enable_lfu:
|
|
||||||
self.lfu = fu_class(out_channels // 2, out_channels // 2, groups)
|
|
||||||
self.conv2 = torch.nn.Conv2d(
|
|
||||||
out_channels // 2, out_channels, kernel_size=1, groups=groups, bias=False
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
x = self.downsample(x)
|
|
||||||
x = self.conv1(x)
|
|
||||||
output = self.fu(x)
|
|
||||||
|
|
||||||
if self.enable_lfu:
|
|
||||||
_, c, h, _ = x.shape
|
|
||||||
split_no = 2
|
|
||||||
split_s = h // split_no
|
|
||||||
xs = torch.cat(
|
|
||||||
torch.split(x[:, : c // 4], split_s, dim=-2), dim=1
|
|
||||||
).contiguous()
|
|
||||||
xs = torch.cat(torch.split(xs, split_s, dim=-1), dim=1).contiguous()
|
|
||||||
xs = self.lfu(xs)
|
|
||||||
xs = xs.repeat(1, 1, split_no, split_no).contiguous()
|
|
||||||
else:
|
|
||||||
xs = 0
|
|
||||||
|
|
||||||
output = self.conv2(x + output + xs)
|
|
||||||
|
|
||||||
return output
|
|
||||||
|
|
||||||
|
|
||||||
class FFC(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
in_channels,
|
|
||||||
out_channels,
|
|
||||||
kernel_size,
|
|
||||||
ratio_gin,
|
|
||||||
ratio_gout,
|
|
||||||
stride=1,
|
|
||||||
padding=0,
|
|
||||||
dilation=1,
|
|
||||||
groups=1,
|
|
||||||
bias=False,
|
|
||||||
enable_lfu=True,
|
|
||||||
padding_type="reflect",
|
|
||||||
gated=False,
|
|
||||||
**spectral_kwargs,
|
|
||||||
):
|
|
||||||
super(FFC, self).__init__()
|
|
||||||
|
|
||||||
assert stride == 1 or stride == 2, "Stride should be 1 or 2."
|
|
||||||
self.stride = stride
|
|
||||||
|
|
||||||
in_cg = int(in_channels * ratio_gin)
|
|
||||||
in_cl = in_channels - in_cg
|
|
||||||
out_cg = int(out_channels * ratio_gout)
|
|
||||||
out_cl = out_channels - out_cg
|
|
||||||
# groups_g = 1 if groups == 1 else int(groups * ratio_gout)
|
|
||||||
# groups_l = 1 if groups == 1 else groups - groups_g
|
|
||||||
|
|
||||||
self.ratio_gin = ratio_gin
|
|
||||||
self.ratio_gout = ratio_gout
|
|
||||||
self.global_in_num = in_cg
|
|
||||||
|
|
||||||
module = nn.Identity if in_cl == 0 or out_cl == 0 else nn.Conv2d
|
|
||||||
self.convl2l = module(
|
|
||||||
in_cl,
|
|
||||||
out_cl,
|
|
||||||
kernel_size,
|
|
||||||
stride,
|
|
||||||
padding,
|
|
||||||
dilation,
|
|
||||||
groups,
|
|
||||||
bias,
|
|
||||||
padding_mode=padding_type,
|
|
||||||
)
|
|
||||||
module = nn.Identity if in_cl == 0 or out_cg == 0 else nn.Conv2d
|
|
||||||
self.convl2g = module(
|
|
||||||
in_cl,
|
|
||||||
out_cg,
|
|
||||||
kernel_size,
|
|
||||||
stride,
|
|
||||||
padding,
|
|
||||||
dilation,
|
|
||||||
groups,
|
|
||||||
bias,
|
|
||||||
padding_mode=padding_type,
|
|
||||||
)
|
|
||||||
module = nn.Identity if in_cg == 0 or out_cl == 0 else nn.Conv2d
|
|
||||||
self.convg2l = module(
|
|
||||||
in_cg,
|
|
||||||
out_cl,
|
|
||||||
kernel_size,
|
|
||||||
stride,
|
|
||||||
padding,
|
|
||||||
dilation,
|
|
||||||
groups,
|
|
||||||
bias,
|
|
||||||
padding_mode=padding_type,
|
|
||||||
)
|
|
||||||
module = nn.Identity if in_cg == 0 or out_cg == 0 else SpectralTransform
|
|
||||||
self.convg2g = module(
|
|
||||||
in_cg,
|
|
||||||
out_cg,
|
|
||||||
stride,
|
|
||||||
1 if groups == 1 else groups // 2,
|
|
||||||
enable_lfu,
|
|
||||||
**spectral_kwargs,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.gated = gated
|
|
||||||
module = (
|
|
||||||
nn.Identity if in_cg == 0 or out_cl == 0 or not self.gated else nn.Conv2d
|
|
||||||
)
|
|
||||||
self.gate = module(in_channels, 2, 1)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
x_l, x_g = x if type(x) is tuple else (x, 0)
|
|
||||||
out_xl, out_xg = 0, 0
|
|
||||||
|
|
||||||
if self.gated:
|
|
||||||
total_input_parts = [x_l]
|
|
||||||
if torch.is_tensor(x_g):
|
|
||||||
total_input_parts.append(x_g)
|
|
||||||
total_input = torch.cat(total_input_parts, dim=1)
|
|
||||||
|
|
||||||
gates = torch.sigmoid(self.gate(total_input))
|
|
||||||
g2l_gate, l2g_gate = gates.chunk(2, dim=1)
|
|
||||||
else:
|
|
||||||
g2l_gate, l2g_gate = 1, 1
|
|
||||||
|
|
||||||
if self.ratio_gout != 1:
|
|
||||||
out_xl = self.convl2l(x_l) + self.convg2l(x_g) * g2l_gate
|
|
||||||
if self.ratio_gout != 0:
|
|
||||||
out_xg = self.convl2g(x_l) * l2g_gate + self.convg2g(x_g)
|
|
||||||
|
|
||||||
return out_xl, out_xg
|
|
||||||
|
|
||||||
|
|
||||||
class FFC_BN_ACT(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
in_channels,
|
|
||||||
out_channels,
|
|
||||||
kernel_size,
|
|
||||||
ratio_gin,
|
|
||||||
ratio_gout,
|
|
||||||
stride=1,
|
|
||||||
padding=0,
|
|
||||||
dilation=1,
|
|
||||||
groups=1,
|
|
||||||
bias=False,
|
|
||||||
norm_layer=nn.BatchNorm2d,
|
|
||||||
activation_layer=nn.Identity,
|
|
||||||
padding_type="reflect",
|
|
||||||
enable_lfu=True,
|
|
||||||
**kwargs,
|
|
||||||
):
|
|
||||||
super(FFC_BN_ACT, self).__init__()
|
|
||||||
self.ffc = FFC(
|
|
||||||
in_channels,
|
|
||||||
out_channels,
|
|
||||||
kernel_size,
|
|
||||||
ratio_gin,
|
|
||||||
ratio_gout,
|
|
||||||
stride,
|
|
||||||
padding,
|
|
||||||
dilation,
|
|
||||||
groups,
|
|
||||||
bias,
|
|
||||||
enable_lfu,
|
|
||||||
padding_type=padding_type,
|
|
||||||
**kwargs,
|
|
||||||
)
|
|
||||||
lnorm = nn.Identity if ratio_gout == 1 else norm_layer
|
|
||||||
gnorm = nn.Identity if ratio_gout == 0 else norm_layer
|
|
||||||
global_channels = int(out_channels * ratio_gout)
|
|
||||||
self.bn_l = lnorm(out_channels - global_channels)
|
|
||||||
self.bn_g = gnorm(global_channels)
|
|
||||||
|
|
||||||
lact = nn.Identity if ratio_gout == 1 else activation_layer
|
|
||||||
gact = nn.Identity if ratio_gout == 0 else activation_layer
|
|
||||||
self.act_l = lact(inplace=True)
|
|
||||||
self.act_g = gact(inplace=True)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
x_l, x_g = self.ffc(x)
|
|
||||||
x_l = self.act_l(self.bn_l(x_l))
|
|
||||||
x_g = self.act_g(self.bn_g(x_g))
|
|
||||||
return x_l, x_g
|
|
||||||
|
|
||||||
|
|
||||||
class FFCResnetBlock(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
dim,
|
|
||||||
padding_type,
|
|
||||||
norm_layer,
|
|
||||||
activation_layer=nn.ReLU,
|
|
||||||
dilation=1,
|
|
||||||
spatial_transform_kwargs=None,
|
|
||||||
inline=False,
|
|
||||||
**conv_kwargs,
|
|
||||||
):
|
|
||||||
super().__init__()
|
|
||||||
self.conv1 = FFC_BN_ACT(
|
|
||||||
dim,
|
|
||||||
dim,
|
|
||||||
kernel_size=3,
|
|
||||||
padding=dilation,
|
|
||||||
dilation=dilation,
|
|
||||||
norm_layer=norm_layer,
|
|
||||||
activation_layer=activation_layer,
|
|
||||||
padding_type=padding_type,
|
|
||||||
**conv_kwargs,
|
|
||||||
)
|
|
||||||
self.conv2 = FFC_BN_ACT(
|
|
||||||
dim,
|
|
||||||
dim,
|
|
||||||
kernel_size=3,
|
|
||||||
padding=dilation,
|
|
||||||
dilation=dilation,
|
|
||||||
norm_layer=norm_layer,
|
|
||||||
activation_layer=activation_layer,
|
|
||||||
padding_type=padding_type,
|
|
||||||
**conv_kwargs,
|
|
||||||
)
|
|
||||||
if spatial_transform_kwargs is not None:
|
|
||||||
self.conv1 = LearnableSpatialTransformWrapper(
|
|
||||||
self.conv1, **spatial_transform_kwargs
|
|
||||||
)
|
|
||||||
self.conv2 = LearnableSpatialTransformWrapper(
|
|
||||||
self.conv2, **spatial_transform_kwargs
|
|
||||||
)
|
|
||||||
self.inline = inline
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
if self.inline:
|
|
||||||
x_l, x_g = (
|
|
||||||
x[:, : -self.conv1.ffc.global_in_num],
|
|
||||||
x[:, -self.conv1.ffc.global_in_num :],
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
x_l, x_g = x if type(x) is tuple else (x, 0)
|
|
||||||
|
|
||||||
id_l, id_g = x_l, x_g
|
|
||||||
|
|
||||||
x_l, x_g = self.conv1((x_l, x_g))
|
|
||||||
x_l, x_g = self.conv2((x_l, x_g))
|
|
||||||
|
|
||||||
x_l, x_g = id_l + x_l, id_g + x_g
|
|
||||||
out = x_l, x_g
|
|
||||||
if self.inline:
|
|
||||||
out = torch.cat(out, dim=1)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
class ConcatTupleLayer(nn.Module):
|
|
||||||
def forward(self, x):
|
|
||||||
assert isinstance(x, tuple)
|
|
||||||
x_l, x_g = x
|
|
||||||
assert torch.is_tensor(x_l) or torch.is_tensor(x_g)
|
|
||||||
if not torch.is_tensor(x_g):
|
|
||||||
return x_l
|
|
||||||
return torch.cat(x, dim=1)
|
|
||||||
|
|
||||||
|
|
||||||
class FFCResNetGenerator(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
input_nc,
|
|
||||||
output_nc,
|
|
||||||
ngf=64,
|
|
||||||
n_downsampling=3,
|
|
||||||
n_blocks=18,
|
|
||||||
norm_layer=nn.BatchNorm2d,
|
|
||||||
padding_type="reflect",
|
|
||||||
activation_layer=nn.ReLU,
|
|
||||||
up_norm_layer=nn.BatchNorm2d,
|
|
||||||
up_activation=nn.ReLU(True),
|
|
||||||
init_conv_kwargs={},
|
|
||||||
downsample_conv_kwargs={},
|
|
||||||
resnet_conv_kwargs={},
|
|
||||||
spatial_transform_layers=None,
|
|
||||||
spatial_transform_kwargs={},
|
|
||||||
max_features=1024,
|
|
||||||
out_ffc=False,
|
|
||||||
out_ffc_kwargs={},
|
|
||||||
):
|
|
||||||
assert n_blocks >= 0
|
|
||||||
super().__init__()
|
|
||||||
"""
|
|
||||||
init_conv_kwargs = {'ratio_gin': 0, 'ratio_gout': 0, 'enable_lfu': False}
|
|
||||||
downsample_conv_kwargs = {'ratio_gin': '${generator.init_conv_kwargs.ratio_gout}', 'ratio_gout': '${generator.downsample_conv_kwargs.ratio_gin}', 'enable_lfu': False}
|
|
||||||
resnet_conv_kwargs = {'ratio_gin': 0.75, 'ratio_gout': '${generator.resnet_conv_kwargs.ratio_gin}', 'enable_lfu': False}
|
|
||||||
spatial_transform_kwargs = {}
|
|
||||||
out_ffc_kwargs = {}
|
|
||||||
"""
|
|
||||||
"""
|
|
||||||
print(input_nc, output_nc, ngf, n_downsampling, n_blocks, norm_layer,
|
|
||||||
padding_type, activation_layer,
|
|
||||||
up_norm_layer, up_activation,
|
|
||||||
spatial_transform_layers,
|
|
||||||
add_out_act, max_features, out_ffc, file=sys.stderr)
|
|
||||||
|
|
||||||
4 3 64 3 18 <class 'torch.nn.modules.batchnorm.BatchNorm2d'>
|
|
||||||
reflect <class 'torch.nn.modules.activation.ReLU'>
|
|
||||||
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>
|
|
||||||
ReLU(inplace=True)
|
|
||||||
None sigmoid 1024 False
|
|
||||||
"""
|
|
||||||
init_conv_kwargs = {"ratio_gin": 0, "ratio_gout": 0, "enable_lfu": False}
|
|
||||||
downsample_conv_kwargs = {"ratio_gin": 0, "ratio_gout": 0, "enable_lfu": False}
|
|
||||||
resnet_conv_kwargs = {
|
|
||||||
"ratio_gin": 0.75,
|
|
||||||
"ratio_gout": 0.75,
|
|
||||||
"enable_lfu": False,
|
|
||||||
}
|
|
||||||
spatial_transform_kwargs = {}
|
|
||||||
out_ffc_kwargs = {}
|
|
||||||
|
|
||||||
model = [
|
|
||||||
nn.ReflectionPad2d(3),
|
|
||||||
FFC_BN_ACT(
|
|
||||||
input_nc,
|
|
||||||
ngf,
|
|
||||||
kernel_size=7,
|
|
||||||
padding=0,
|
|
||||||
norm_layer=norm_layer,
|
|
||||||
activation_layer=activation_layer,
|
|
||||||
**init_conv_kwargs,
|
|
||||||
),
|
|
||||||
]
|
|
||||||
|
|
||||||
### downsample
|
|
||||||
for i in range(n_downsampling):
|
|
||||||
mult = 2**i
|
|
||||||
if i == n_downsampling - 1:
|
|
||||||
cur_conv_kwargs = dict(downsample_conv_kwargs)
|
|
||||||
cur_conv_kwargs["ratio_gout"] = resnet_conv_kwargs.get("ratio_gin", 0)
|
|
||||||
else:
|
|
||||||
cur_conv_kwargs = downsample_conv_kwargs
|
|
||||||
model += [
|
|
||||||
FFC_BN_ACT(
|
|
||||||
min(max_features, ngf * mult),
|
|
||||||
min(max_features, ngf * mult * 2),
|
|
||||||
kernel_size=3,
|
|
||||||
stride=2,
|
|
||||||
padding=1,
|
|
||||||
norm_layer=norm_layer,
|
|
||||||
activation_layer=activation_layer,
|
|
||||||
**cur_conv_kwargs,
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
mult = 2**n_downsampling
|
|
||||||
feats_num_bottleneck = min(max_features, ngf * mult)
|
|
||||||
|
|
||||||
### resnet blocks
|
|
||||||
for i in range(n_blocks):
|
|
||||||
cur_resblock = FFCResnetBlock(
|
|
||||||
feats_num_bottleneck,
|
|
||||||
padding_type=padding_type,
|
|
||||||
activation_layer=activation_layer,
|
|
||||||
norm_layer=norm_layer,
|
|
||||||
**resnet_conv_kwargs,
|
|
||||||
)
|
|
||||||
if spatial_transform_layers is not None and i in spatial_transform_layers:
|
|
||||||
cur_resblock = LearnableSpatialTransformWrapper(
|
|
||||||
cur_resblock, **spatial_transform_kwargs
|
|
||||||
)
|
|
||||||
model += [cur_resblock]
|
|
||||||
|
|
||||||
model += [ConcatTupleLayer()]
|
|
||||||
|
|
||||||
### upsample
|
|
||||||
for i in range(n_downsampling):
|
|
||||||
mult = 2 ** (n_downsampling - i)
|
|
||||||
model += [
|
|
||||||
nn.ConvTranspose2d(
|
|
||||||
min(max_features, ngf * mult),
|
|
||||||
min(max_features, int(ngf * mult / 2)),
|
|
||||||
kernel_size=3,
|
|
||||||
stride=2,
|
|
||||||
padding=1,
|
|
||||||
output_padding=1,
|
|
||||||
),
|
|
||||||
up_norm_layer(min(max_features, int(ngf * mult / 2))),
|
|
||||||
up_activation,
|
|
||||||
]
|
|
||||||
|
|
||||||
if out_ffc:
|
|
||||||
model += [
|
|
||||||
FFCResnetBlock(
|
|
||||||
ngf,
|
|
||||||
padding_type=padding_type,
|
|
||||||
activation_layer=activation_layer,
|
|
||||||
norm_layer=norm_layer,
|
|
||||||
inline=True,
|
|
||||||
**out_ffc_kwargs,
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
model += [
|
|
||||||
nn.ReflectionPad2d(3),
|
|
||||||
nn.Conv2d(ngf, output_nc, kernel_size=7, padding=0),
|
|
||||||
]
|
|
||||||
model.append(nn.Sigmoid())
|
|
||||||
self.model = nn.Sequential(*model)
|
|
||||||
|
|
||||||
def forward(self, image, mask):
|
|
||||||
return self.model(torch.cat([image, mask], dim=1))
|
|
||||||
|
|
||||||
|
|
||||||
class LaMa(nn.Module):
|
|
||||||
def __init__(self, state_dict) -> None:
|
|
||||||
super(LaMa, self).__init__()
|
|
||||||
self.model_arch = "LaMa"
|
|
||||||
self.sub_type = "Inpaint"
|
|
||||||
self.in_nc = 4
|
|
||||||
self.out_nc = 3
|
|
||||||
self.scale = 1
|
|
||||||
|
|
||||||
self.min_size = None
|
|
||||||
self.pad_mod = 8
|
|
||||||
self.pad_to_square = False
|
|
||||||
|
|
||||||
self.model = FFCResNetGenerator(self.in_nc, self.out_nc)
|
|
||||||
self.state = {
|
|
||||||
k.replace("generator.model", "model.model"): v
|
|
||||||
for k, v in state_dict.items()
|
|
||||||
}
|
|
||||||
|
|
||||||
self.supports_fp16 = False
|
|
||||||
self.support_bf16 = True
|
|
||||||
|
|
||||||
self.load_state_dict(self.state, strict=False)
|
|
||||||
|
|
||||||
def forward(self, img, mask):
|
|
||||||
masked_img = img * (1 - mask)
|
|
||||||
inpainted_mask = mask * self.model.forward(masked_img, mask)
|
|
||||||
result = inpainted_mask + (1 - mask) * img
|
|
||||||
return result
|
|
||||||
@ -1,110 +0,0 @@
|
|||||||
import math
|
|
||||||
|
|
||||||
import torch.nn as nn
|
|
||||||
|
|
||||||
|
|
||||||
class CA_layer(nn.Module):
|
|
||||||
def __init__(self, channel, reduction=16):
|
|
||||||
super(CA_layer, self).__init__()
|
|
||||||
# global average pooling
|
|
||||||
self.gap = nn.AdaptiveAvgPool2d(1)
|
|
||||||
self.fc = nn.Sequential(
|
|
||||||
nn.Conv2d(channel, channel // reduction, kernel_size=(1, 1), bias=False),
|
|
||||||
nn.GELU(),
|
|
||||||
nn.Conv2d(channel // reduction, channel, kernel_size=(1, 1), bias=False),
|
|
||||||
# nn.Sigmoid()
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
y = self.fc(self.gap(x))
|
|
||||||
return x * y.expand_as(x)
|
|
||||||
|
|
||||||
|
|
||||||
class Simple_CA_layer(nn.Module):
|
|
||||||
def __init__(self, channel):
|
|
||||||
super(Simple_CA_layer, self).__init__()
|
|
||||||
self.gap = nn.AdaptiveAvgPool2d(1)
|
|
||||||
self.fc = nn.Conv2d(
|
|
||||||
in_channels=channel,
|
|
||||||
out_channels=channel,
|
|
||||||
kernel_size=1,
|
|
||||||
padding=0,
|
|
||||||
stride=1,
|
|
||||||
groups=1,
|
|
||||||
bias=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return x * self.fc(self.gap(x))
|
|
||||||
|
|
||||||
|
|
||||||
class ECA_layer(nn.Module):
|
|
||||||
"""Constructs a ECA module.
|
|
||||||
Args:
|
|
||||||
channel: Number of channels of the input feature map
|
|
||||||
k_size: Adaptive selection of kernel size
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, channel):
|
|
||||||
super(ECA_layer, self).__init__()
|
|
||||||
|
|
||||||
b = 1
|
|
||||||
gamma = 2
|
|
||||||
k_size = int(abs(math.log(channel, 2) + b) / gamma)
|
|
||||||
k_size = k_size if k_size % 2 else k_size + 1
|
|
||||||
self.avg_pool = nn.AdaptiveAvgPool2d(1)
|
|
||||||
self.conv = nn.Conv1d(
|
|
||||||
1, 1, kernel_size=k_size, padding=(k_size - 1) // 2, bias=False
|
|
||||||
)
|
|
||||||
# self.sigmoid = nn.Sigmoid()
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
# x: input features with shape [b, c, h, w]
|
|
||||||
# b, c, h, w = x.size()
|
|
||||||
|
|
||||||
# feature descriptor on the global spatial information
|
|
||||||
y = self.avg_pool(x)
|
|
||||||
|
|
||||||
# Two different branches of ECA module
|
|
||||||
y = self.conv(y.squeeze(-1).transpose(-1, -2)).transpose(-1, -2).unsqueeze(-1)
|
|
||||||
|
|
||||||
# Multi-scale information fusion
|
|
||||||
# y = self.sigmoid(y)
|
|
||||||
|
|
||||||
return x * y.expand_as(x)
|
|
||||||
|
|
||||||
|
|
||||||
class ECA_MaxPool_layer(nn.Module):
|
|
||||||
"""Constructs a ECA module.
|
|
||||||
Args:
|
|
||||||
channel: Number of channels of the input feature map
|
|
||||||
k_size: Adaptive selection of kernel size
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, channel):
|
|
||||||
super(ECA_MaxPool_layer, self).__init__()
|
|
||||||
|
|
||||||
b = 1
|
|
||||||
gamma = 2
|
|
||||||
k_size = int(abs(math.log(channel, 2) + b) / gamma)
|
|
||||||
k_size = k_size if k_size % 2 else k_size + 1
|
|
||||||
self.max_pool = nn.AdaptiveMaxPool2d(1)
|
|
||||||
self.conv = nn.Conv1d(
|
|
||||||
1, 1, kernel_size=k_size, padding=(k_size - 1) // 2, bias=False
|
|
||||||
)
|
|
||||||
# self.sigmoid = nn.Sigmoid()
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
# x: input features with shape [b, c, h, w]
|
|
||||||
# b, c, h, w = x.size()
|
|
||||||
|
|
||||||
# feature descriptor on the global spatial information
|
|
||||||
y = self.max_pool(x)
|
|
||||||
|
|
||||||
# Two different branches of ECA module
|
|
||||||
y = self.conv(y.squeeze(-1).transpose(-1, -2)).transpose(-1, -2).unsqueeze(-1)
|
|
||||||
|
|
||||||
# Multi-scale information fusion
|
|
||||||
# y = self.sigmoid(y)
|
|
||||||
|
|
||||||
return x * y.expand_as(x)
|
|
||||||
@ -1,201 +0,0 @@
|
|||||||
Apache License
|
|
||||||
Version 2.0, January 2004
|
|
||||||
http://www.apache.org/licenses/
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
||||||
|
|
||||||
1. Definitions.
|
|
||||||
|
|
||||||
"License" shall mean the terms and conditions for use, reproduction,
|
|
||||||
and distribution as defined by Sections 1 through 9 of this document.
|
|
||||||
|
|
||||||
"Licensor" shall mean the copyright owner or entity authorized by
|
|
||||||
the copyright owner that is granting the License.
|
|
||||||
|
|
||||||
"Legal Entity" shall mean the union of the acting entity and all
|
|
||||||
other entities that control, are controlled by, or are under common
|
|
||||||
control with that entity. For the purposes of this definition,
|
|
||||||
"control" means (i) the power, direct or indirect, to cause the
|
|
||||||
direction or management of such entity, whether by contract or
|
|
||||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
||||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
||||||
|
|
||||||
"You" (or "Your") shall mean an individual or Legal Entity
|
|
||||||
exercising permissions granted by this License.
|
|
||||||
|
|
||||||
"Source" form shall mean the preferred form for making modifications,
|
|
||||||
including but not limited to software source code, documentation
|
|
||||||
source, and configuration files.
|
|
||||||
|
|
||||||
"Object" form shall mean any form resulting from mechanical
|
|
||||||
transformation or translation of a Source form, including but
|
|
||||||
not limited to compiled object code, generated documentation,
|
|
||||||
and conversions to other media types.
|
|
||||||
|
|
||||||
"Work" shall mean the work of authorship, whether in Source or
|
|
||||||
Object form, made available under the License, as indicated by a
|
|
||||||
copyright notice that is included in or attached to the work
|
|
||||||
(an example is provided in the Appendix below).
|
|
||||||
|
|
||||||
"Derivative Works" shall mean any work, whether in Source or Object
|
|
||||||
form, that is based on (or derived from) the Work and for which the
|
|
||||||
editorial revisions, annotations, elaborations, or other modifications
|
|
||||||
represent, as a whole, an original work of authorship. For the purposes
|
|
||||||
of this License, Derivative Works shall not include works that remain
|
|
||||||
separable from, or merely link (or bind by name) to the interfaces of,
|
|
||||||
the Work and Derivative Works thereof.
|
|
||||||
|
|
||||||
"Contribution" shall mean any work of authorship, including
|
|
||||||
the original version of the Work and any modifications or additions
|
|
||||||
to that Work or Derivative Works thereof, that is intentionally
|
|
||||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
||||||
or by an individual or Legal Entity authorized to submit on behalf of
|
|
||||||
the copyright owner. For the purposes of this definition, "submitted"
|
|
||||||
means any form of electronic, verbal, or written communication sent
|
|
||||||
to the Licensor or its representatives, including but not limited to
|
|
||||||
communication on electronic mailing lists, source code control systems,
|
|
||||||
and issue tracking systems that are managed by, or on behalf of, the
|
|
||||||
Licensor for the purpose of discussing and improving the Work, but
|
|
||||||
excluding communication that is conspicuously marked or otherwise
|
|
||||||
designated in writing by the copyright owner as "Not a Contribution."
|
|
||||||
|
|
||||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
||||||
on behalf of whom a Contribution has been received by Licensor and
|
|
||||||
subsequently incorporated within the Work.
|
|
||||||
|
|
||||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
copyright license to reproduce, prepare Derivative Works of,
|
|
||||||
publicly display, publicly perform, sublicense, and distribute the
|
|
||||||
Work and such Derivative Works in Source or Object form.
|
|
||||||
|
|
||||||
3. Grant of Patent License. Subject to the terms and conditions of
|
|
||||||
this License, each Contributor hereby grants to You a perpetual,
|
|
||||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
||||||
(except as stated in this section) patent license to make, have made,
|
|
||||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
||||||
where such license applies only to those patent claims licensable
|
|
||||||
by such Contributor that are necessarily infringed by their
|
|
||||||
Contribution(s) alone or by combination of their Contribution(s)
|
|
||||||
with the Work to which such Contribution(s) was submitted. If You
|
|
||||||
institute patent litigation against any entity (including a
|
|
||||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
||||||
or a Contribution incorporated within the Work constitutes direct
|
|
||||||
or contributory patent infringement, then any patent licenses
|
|
||||||
granted to You under this License for that Work shall terminate
|
|
||||||
as of the date such litigation is filed.
|
|
||||||
|
|
||||||
4. Redistribution. You may reproduce and distribute copies of the
|
|
||||||
Work or Derivative Works thereof in any medium, with or without
|
|
||||||
modifications, and in Source or Object form, provided that You
|
|
||||||
meet the following conditions:
|
|
||||||
|
|
||||||
(a) You must give any other recipients of the Work or
|
|
||||||
Derivative Works a copy of this License; and
|
|
||||||
|
|
||||||
(b) You must cause any modified files to carry prominent notices
|
|
||||||
stating that You changed the files; and
|
|
||||||
|
|
||||||
(c) You must retain, in the Source form of any Derivative Works
|
|
||||||
that You distribute, all copyright, patent, trademark, and
|
|
||||||
attribution notices from the Source form of the Work,
|
|
||||||
excluding those notices that do not pertain to any part of
|
|
||||||
the Derivative Works; and
|
|
||||||
|
|
||||||
(d) If the Work includes a "NOTICE" text file as part of its
|
|
||||||
distribution, then any Derivative Works that You distribute must
|
|
||||||
include a readable copy of the attribution notices contained
|
|
||||||
within such NOTICE file, excluding those notices that do not
|
|
||||||
pertain to any part of the Derivative Works, in at least one
|
|
||||||
of the following places: within a NOTICE text file distributed
|
|
||||||
as part of the Derivative Works; within the Source form or
|
|
||||||
documentation, if provided along with the Derivative Works; or,
|
|
||||||
within a display generated by the Derivative Works, if and
|
|
||||||
wherever such third-party notices normally appear. The contents
|
|
||||||
of the NOTICE file are for informational purposes only and
|
|
||||||
do not modify the License. You may add Your own attribution
|
|
||||||
notices within Derivative Works that You distribute, alongside
|
|
||||||
or as an addendum to the NOTICE text from the Work, provided
|
|
||||||
that such additional attribution notices cannot be construed
|
|
||||||
as modifying the License.
|
|
||||||
|
|
||||||
You may add Your own copyright statement to Your modifications and
|
|
||||||
may provide additional or different license terms and conditions
|
|
||||||
for use, reproduction, or distribution of Your modifications, or
|
|
||||||
for any such Derivative Works as a whole, provided Your use,
|
|
||||||
reproduction, and distribution of the Work otherwise complies with
|
|
||||||
the conditions stated in this License.
|
|
||||||
|
|
||||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
||||||
any Contribution intentionally submitted for inclusion in the Work
|
|
||||||
by You to the Licensor shall be under the terms and conditions of
|
|
||||||
this License, without any additional terms or conditions.
|
|
||||||
Notwithstanding the above, nothing herein shall supersede or modify
|
|
||||||
the terms of any separate license agreement you may have executed
|
|
||||||
with Licensor regarding such Contributions.
|
|
||||||
|
|
||||||
6. Trademarks. This License does not grant permission to use the trade
|
|
||||||
names, trademarks, service marks, or product names of the Licensor,
|
|
||||||
except as required for reasonable and customary use in describing the
|
|
||||||
origin of the Work and reproducing the content of the NOTICE file.
|
|
||||||
|
|
||||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
||||||
agreed to in writing, Licensor provides the Work (and each
|
|
||||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
||||||
implied, including, without limitation, any warranties or conditions
|
|
||||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
||||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
||||||
appropriateness of using or redistributing the Work and assume any
|
|
||||||
risks associated with Your exercise of permissions under this License.
|
|
||||||
|
|
||||||
8. Limitation of Liability. In no event and under no legal theory,
|
|
||||||
whether in tort (including negligence), contract, or otherwise,
|
|
||||||
unless required by applicable law (such as deliberate and grossly
|
|
||||||
negligent acts) or agreed to in writing, shall any Contributor be
|
|
||||||
liable to You for damages, including any direct, indirect, special,
|
|
||||||
incidental, or consequential damages of any character arising as a
|
|
||||||
result of this License or out of the use or inability to use the
|
|
||||||
Work (including but not limited to damages for loss of goodwill,
|
|
||||||
work stoppage, computer failure or malfunction, or any and all
|
|
||||||
other commercial damages or losses), even if such Contributor
|
|
||||||
has been advised of the possibility of such damages.
|
|
||||||
|
|
||||||
9. Accepting Warranty or Additional Liability. While redistributing
|
|
||||||
the Work or Derivative Works thereof, You may choose to offer,
|
|
||||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
||||||
or other liability obligations and/or rights consistent with this
|
|
||||||
License. However, in accepting such obligations, You may act only
|
|
||||||
on Your own behalf and on Your sole responsibility, not on behalf
|
|
||||||
of any other Contributor, and only if You agree to indemnify,
|
|
||||||
defend, and hold each Contributor harmless for any liability
|
|
||||||
incurred by, or claims asserted against, such Contributor by reason
|
|
||||||
of your accepting any such warranty or additional liability.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
APPENDIX: How to apply the Apache License to your work.
|
|
||||||
|
|
||||||
To apply the Apache License to your work, attach the following
|
|
||||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
||||||
replaced with your own identifying information. (Don't include
|
|
||||||
the brackets!) The text should be enclosed in the appropriate
|
|
||||||
comment syntax for the file format. We also recommend that a
|
|
||||||
file or class name and description of purpose be included on the
|
|
||||||
same "printed page" as the copyright notice for easier
|
|
||||||
identification within third-party archives.
|
|
||||||
|
|
||||||
Copyright [yyyy] [name of copyright owner]
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
@ -1,577 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding:utf-8 -*-
|
|
||||||
#############################################################
|
|
||||||
# File: OSA.py
|
|
||||||
# Created Date: Tuesday April 28th 2022
|
|
||||||
# Author: Chen Xuanhong
|
|
||||||
# Email: chenxuanhongzju@outlook.com
|
|
||||||
# Last Modified: Sunday, 23rd April 2023 3:07:42 pm
|
|
||||||
# Modified By: Chen Xuanhong
|
|
||||||
# Copyright (c) 2020 Shanghai Jiao Tong University
|
|
||||||
#############################################################
|
|
||||||
|
|
||||||
import torch
|
|
||||||
import torch.nn.functional as F
|
|
||||||
from einops import rearrange, repeat
|
|
||||||
from einops.layers.torch import Rearrange, Reduce
|
|
||||||
from torch import einsum, nn
|
|
||||||
|
|
||||||
from .layernorm import LayerNorm2d
|
|
||||||
|
|
||||||
# helpers
|
|
||||||
|
|
||||||
|
|
||||||
def exists(val):
|
|
||||||
return val is not None
|
|
||||||
|
|
||||||
|
|
||||||
def default(val, d):
|
|
||||||
return val if exists(val) else d
|
|
||||||
|
|
||||||
|
|
||||||
def cast_tuple(val, length=1):
|
|
||||||
return val if isinstance(val, tuple) else ((val,) * length)
|
|
||||||
|
|
||||||
|
|
||||||
# helper classes
|
|
||||||
|
|
||||||
|
|
||||||
class PreNormResidual(nn.Module):
|
|
||||||
def __init__(self, dim, fn):
|
|
||||||
super().__init__()
|
|
||||||
self.norm = nn.LayerNorm(dim)
|
|
||||||
self.fn = fn
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return self.fn(self.norm(x)) + x
|
|
||||||
|
|
||||||
|
|
||||||
class Conv_PreNormResidual(nn.Module):
|
|
||||||
def __init__(self, dim, fn):
|
|
||||||
super().__init__()
|
|
||||||
self.norm = LayerNorm2d(dim)
|
|
||||||
self.fn = fn
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return self.fn(self.norm(x)) + x
|
|
||||||
|
|
||||||
|
|
||||||
class FeedForward(nn.Module):
|
|
||||||
def __init__(self, dim, mult=2, dropout=0.0):
|
|
||||||
super().__init__()
|
|
||||||
inner_dim = int(dim * mult)
|
|
||||||
self.net = nn.Sequential(
|
|
||||||
nn.Linear(dim, inner_dim),
|
|
||||||
nn.GELU(),
|
|
||||||
nn.Dropout(dropout),
|
|
||||||
nn.Linear(inner_dim, dim),
|
|
||||||
nn.Dropout(dropout),
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return self.net(x)
|
|
||||||
|
|
||||||
|
|
||||||
class Conv_FeedForward(nn.Module):
|
|
||||||
def __init__(self, dim, mult=2, dropout=0.0):
|
|
||||||
super().__init__()
|
|
||||||
inner_dim = int(dim * mult)
|
|
||||||
self.net = nn.Sequential(
|
|
||||||
nn.Conv2d(dim, inner_dim, 1, 1, 0),
|
|
||||||
nn.GELU(),
|
|
||||||
nn.Dropout(dropout),
|
|
||||||
nn.Conv2d(inner_dim, dim, 1, 1, 0),
|
|
||||||
nn.Dropout(dropout),
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return self.net(x)
|
|
||||||
|
|
||||||
|
|
||||||
class Gated_Conv_FeedForward(nn.Module):
|
|
||||||
def __init__(self, dim, mult=1, bias=False, dropout=0.0):
|
|
||||||
super().__init__()
|
|
||||||
|
|
||||||
hidden_features = int(dim * mult)
|
|
||||||
|
|
||||||
self.project_in = nn.Conv2d(dim, hidden_features * 2, kernel_size=1, bias=bias)
|
|
||||||
|
|
||||||
self.dwconv = nn.Conv2d(
|
|
||||||
hidden_features * 2,
|
|
||||||
hidden_features * 2,
|
|
||||||
kernel_size=3,
|
|
||||||
stride=1,
|
|
||||||
padding=1,
|
|
||||||
groups=hidden_features * 2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.project_out = nn.Conv2d(hidden_features, dim, kernel_size=1, bias=bias)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
x = self.project_in(x)
|
|
||||||
x1, x2 = self.dwconv(x).chunk(2, dim=1)
|
|
||||||
x = F.gelu(x1) * x2
|
|
||||||
x = self.project_out(x)
|
|
||||||
return x
|
|
||||||
|
|
||||||
|
|
||||||
# MBConv
|
|
||||||
|
|
||||||
|
|
||||||
class SqueezeExcitation(nn.Module):
|
|
||||||
def __init__(self, dim, shrinkage_rate=0.25):
|
|
||||||
super().__init__()
|
|
||||||
hidden_dim = int(dim * shrinkage_rate)
|
|
||||||
|
|
||||||
self.gate = nn.Sequential(
|
|
||||||
Reduce("b c h w -> b c", "mean"),
|
|
||||||
nn.Linear(dim, hidden_dim, bias=False),
|
|
||||||
nn.SiLU(),
|
|
||||||
nn.Linear(hidden_dim, dim, bias=False),
|
|
||||||
nn.Sigmoid(),
|
|
||||||
Rearrange("b c -> b c 1 1"),
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return x * self.gate(x)
|
|
||||||
|
|
||||||
|
|
||||||
class MBConvResidual(nn.Module):
|
|
||||||
def __init__(self, fn, dropout=0.0):
|
|
||||||
super().__init__()
|
|
||||||
self.fn = fn
|
|
||||||
self.dropsample = Dropsample(dropout)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
out = self.fn(x)
|
|
||||||
out = self.dropsample(out)
|
|
||||||
return out + x
|
|
||||||
|
|
||||||
|
|
||||||
class Dropsample(nn.Module):
|
|
||||||
def __init__(self, prob=0):
|
|
||||||
super().__init__()
|
|
||||||
self.prob = prob
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
device = x.device
|
|
||||||
|
|
||||||
if self.prob == 0.0 or (not self.training):
|
|
||||||
return x
|
|
||||||
|
|
||||||
keep_mask = (
|
|
||||||
torch.FloatTensor((x.shape[0], 1, 1, 1), device=device).uniform_()
|
|
||||||
> self.prob
|
|
||||||
)
|
|
||||||
return x * keep_mask / (1 - self.prob)
|
|
||||||
|
|
||||||
|
|
||||||
def MBConv(
|
|
||||||
dim_in, dim_out, *, downsample, expansion_rate=4, shrinkage_rate=0.25, dropout=0.0
|
|
||||||
):
|
|
||||||
hidden_dim = int(expansion_rate * dim_out)
|
|
||||||
stride = 2 if downsample else 1
|
|
||||||
|
|
||||||
net = nn.Sequential(
|
|
||||||
nn.Conv2d(dim_in, hidden_dim, 1),
|
|
||||||
# nn.BatchNorm2d(hidden_dim),
|
|
||||||
nn.GELU(),
|
|
||||||
nn.Conv2d(
|
|
||||||
hidden_dim, hidden_dim, 3, stride=stride, padding=1, groups=hidden_dim
|
|
||||||
),
|
|
||||||
# nn.BatchNorm2d(hidden_dim),
|
|
||||||
nn.GELU(),
|
|
||||||
SqueezeExcitation(hidden_dim, shrinkage_rate=shrinkage_rate),
|
|
||||||
nn.Conv2d(hidden_dim, dim_out, 1),
|
|
||||||
# nn.BatchNorm2d(dim_out)
|
|
||||||
)
|
|
||||||
|
|
||||||
if dim_in == dim_out and not downsample:
|
|
||||||
net = MBConvResidual(net, dropout=dropout)
|
|
||||||
|
|
||||||
return net
|
|
||||||
|
|
||||||
|
|
||||||
# attention related classes
|
|
||||||
class Attention(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
dim,
|
|
||||||
dim_head=32,
|
|
||||||
dropout=0.0,
|
|
||||||
window_size=7,
|
|
||||||
with_pe=True,
|
|
||||||
):
|
|
||||||
super().__init__()
|
|
||||||
assert (
|
|
||||||
dim % dim_head
|
|
||||||
) == 0, "dimension should be divisible by dimension per head"
|
|
||||||
|
|
||||||
self.heads = dim // dim_head
|
|
||||||
self.scale = dim_head**-0.5
|
|
||||||
self.with_pe = with_pe
|
|
||||||
|
|
||||||
self.to_qkv = nn.Linear(dim, dim * 3, bias=False)
|
|
||||||
|
|
||||||
self.attend = nn.Sequential(nn.Softmax(dim=-1), nn.Dropout(dropout))
|
|
||||||
|
|
||||||
self.to_out = nn.Sequential(
|
|
||||||
nn.Linear(dim, dim, bias=False), nn.Dropout(dropout)
|
|
||||||
)
|
|
||||||
|
|
||||||
# relative positional bias
|
|
||||||
if self.with_pe:
|
|
||||||
self.rel_pos_bias = nn.Embedding((2 * window_size - 1) ** 2, self.heads)
|
|
||||||
|
|
||||||
pos = torch.arange(window_size)
|
|
||||||
grid = torch.stack(torch.meshgrid(pos, pos))
|
|
||||||
grid = rearrange(grid, "c i j -> (i j) c")
|
|
||||||
rel_pos = rearrange(grid, "i ... -> i 1 ...") - rearrange(
|
|
||||||
grid, "j ... -> 1 j ..."
|
|
||||||
)
|
|
||||||
rel_pos += window_size - 1
|
|
||||||
rel_pos_indices = (rel_pos * torch.tensor([2 * window_size - 1, 1])).sum(
|
|
||||||
dim=-1
|
|
||||||
)
|
|
||||||
|
|
||||||
self.register_buffer("rel_pos_indices", rel_pos_indices, persistent=False)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
batch, height, width, window_height, window_width, _, device, h = (
|
|
||||||
*x.shape,
|
|
||||||
x.device,
|
|
||||||
self.heads,
|
|
||||||
)
|
|
||||||
|
|
||||||
# flatten
|
|
||||||
|
|
||||||
x = rearrange(x, "b x y w1 w2 d -> (b x y) (w1 w2) d")
|
|
||||||
|
|
||||||
# project for queries, keys, values
|
|
||||||
|
|
||||||
q, k, v = self.to_qkv(x).chunk(3, dim=-1)
|
|
||||||
|
|
||||||
# split heads
|
|
||||||
|
|
||||||
q, k, v = map(lambda t: rearrange(t, "b n (h d ) -> b h n d", h=h), (q, k, v))
|
|
||||||
|
|
||||||
# scale
|
|
||||||
|
|
||||||
q = q * self.scale
|
|
||||||
|
|
||||||
# sim
|
|
||||||
|
|
||||||
sim = einsum("b h i d, b h j d -> b h i j", q, k)
|
|
||||||
|
|
||||||
# add positional bias
|
|
||||||
if self.with_pe:
|
|
||||||
bias = self.rel_pos_bias(self.rel_pos_indices)
|
|
||||||
sim = sim + rearrange(bias, "i j h -> h i j")
|
|
||||||
|
|
||||||
# attention
|
|
||||||
|
|
||||||
attn = self.attend(sim)
|
|
||||||
|
|
||||||
# aggregate
|
|
||||||
|
|
||||||
out = einsum("b h i j, b h j d -> b h i d", attn, v)
|
|
||||||
|
|
||||||
# merge heads
|
|
||||||
|
|
||||||
out = rearrange(
|
|
||||||
out, "b h (w1 w2) d -> b w1 w2 (h d)", w1=window_height, w2=window_width
|
|
||||||
)
|
|
||||||
|
|
||||||
# combine heads out
|
|
||||||
|
|
||||||
out = self.to_out(out)
|
|
||||||
return rearrange(out, "(b x y) ... -> b x y ...", x=height, y=width)
|
|
||||||
|
|
||||||
|
|
||||||
class Block_Attention(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
dim,
|
|
||||||
dim_head=32,
|
|
||||||
bias=False,
|
|
||||||
dropout=0.0,
|
|
||||||
window_size=7,
|
|
||||||
with_pe=True,
|
|
||||||
):
|
|
||||||
super().__init__()
|
|
||||||
assert (
|
|
||||||
dim % dim_head
|
|
||||||
) == 0, "dimension should be divisible by dimension per head"
|
|
||||||
|
|
||||||
self.heads = dim // dim_head
|
|
||||||
self.ps = window_size
|
|
||||||
self.scale = dim_head**-0.5
|
|
||||||
self.with_pe = with_pe
|
|
||||||
|
|
||||||
self.qkv = nn.Conv2d(dim, dim * 3, kernel_size=1, bias=bias)
|
|
||||||
self.qkv_dwconv = nn.Conv2d(
|
|
||||||
dim * 3,
|
|
||||||
dim * 3,
|
|
||||||
kernel_size=3,
|
|
||||||
stride=1,
|
|
||||||
padding=1,
|
|
||||||
groups=dim * 3,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.attend = nn.Sequential(nn.Softmax(dim=-1), nn.Dropout(dropout))
|
|
||||||
|
|
||||||
self.to_out = nn.Conv2d(dim, dim, kernel_size=1, bias=bias)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
# project for queries, keys, values
|
|
||||||
b, c, h, w = x.shape
|
|
||||||
|
|
||||||
qkv = self.qkv_dwconv(self.qkv(x))
|
|
||||||
q, k, v = qkv.chunk(3, dim=1)
|
|
||||||
|
|
||||||
# split heads
|
|
||||||
|
|
||||||
q, k, v = map(
|
|
||||||
lambda t: rearrange(
|
|
||||||
t,
|
|
||||||
"b (h d) (x w1) (y w2) -> (b x y) h (w1 w2) d",
|
|
||||||
h=self.heads,
|
|
||||||
w1=self.ps,
|
|
||||||
w2=self.ps,
|
|
||||||
),
|
|
||||||
(q, k, v),
|
|
||||||
)
|
|
||||||
|
|
||||||
# scale
|
|
||||||
|
|
||||||
q = q * self.scale
|
|
||||||
|
|
||||||
# sim
|
|
||||||
|
|
||||||
sim = einsum("b h i d, b h j d -> b h i j", q, k)
|
|
||||||
|
|
||||||
# attention
|
|
||||||
attn = self.attend(sim)
|
|
||||||
|
|
||||||
# aggregate
|
|
||||||
|
|
||||||
out = einsum("b h i j, b h j d -> b h i d", attn, v)
|
|
||||||
|
|
||||||
# merge heads
|
|
||||||
out = rearrange(
|
|
||||||
out,
|
|
||||||
"(b x y) head (w1 w2) d -> b (head d) (x w1) (y w2)",
|
|
||||||
x=h // self.ps,
|
|
||||||
y=w // self.ps,
|
|
||||||
head=self.heads,
|
|
||||||
w1=self.ps,
|
|
||||||
w2=self.ps,
|
|
||||||
)
|
|
||||||
|
|
||||||
out = self.to_out(out)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
class Channel_Attention(nn.Module):
|
|
||||||
def __init__(self, dim, heads, bias=False, dropout=0.0, window_size=7):
|
|
||||||
super(Channel_Attention, self).__init__()
|
|
||||||
self.heads = heads
|
|
||||||
|
|
||||||
self.temperature = nn.Parameter(torch.ones(heads, 1, 1))
|
|
||||||
|
|
||||||
self.ps = window_size
|
|
||||||
|
|
||||||
self.qkv = nn.Conv2d(dim, dim * 3, kernel_size=1, bias=bias)
|
|
||||||
self.qkv_dwconv = nn.Conv2d(
|
|
||||||
dim * 3,
|
|
||||||
dim * 3,
|
|
||||||
kernel_size=3,
|
|
||||||
stride=1,
|
|
||||||
padding=1,
|
|
||||||
groups=dim * 3,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.project_out = nn.Conv2d(dim, dim, kernel_size=1, bias=bias)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
b, c, h, w = x.shape
|
|
||||||
|
|
||||||
qkv = self.qkv_dwconv(self.qkv(x))
|
|
||||||
qkv = qkv.chunk(3, dim=1)
|
|
||||||
|
|
||||||
q, k, v = map(
|
|
||||||
lambda t: rearrange(
|
|
||||||
t,
|
|
||||||
"b (head d) (h ph) (w pw) -> b (h w) head d (ph pw)",
|
|
||||||
ph=self.ps,
|
|
||||||
pw=self.ps,
|
|
||||||
head=self.heads,
|
|
||||||
),
|
|
||||||
qkv,
|
|
||||||
)
|
|
||||||
|
|
||||||
q = F.normalize(q, dim=-1)
|
|
||||||
k = F.normalize(k, dim=-1)
|
|
||||||
|
|
||||||
attn = (q @ k.transpose(-2, -1)) * self.temperature
|
|
||||||
attn = attn.softmax(dim=-1)
|
|
||||||
out = attn @ v
|
|
||||||
|
|
||||||
out = rearrange(
|
|
||||||
out,
|
|
||||||
"b (h w) head d (ph pw) -> b (head d) (h ph) (w pw)",
|
|
||||||
h=h // self.ps,
|
|
||||||
w=w // self.ps,
|
|
||||||
ph=self.ps,
|
|
||||||
pw=self.ps,
|
|
||||||
head=self.heads,
|
|
||||||
)
|
|
||||||
|
|
||||||
out = self.project_out(out)
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
class Channel_Attention_grid(nn.Module):
|
|
||||||
def __init__(self, dim, heads, bias=False, dropout=0.0, window_size=7):
|
|
||||||
super(Channel_Attention_grid, self).__init__()
|
|
||||||
self.heads = heads
|
|
||||||
|
|
||||||
self.temperature = nn.Parameter(torch.ones(heads, 1, 1))
|
|
||||||
|
|
||||||
self.ps = window_size
|
|
||||||
|
|
||||||
self.qkv = nn.Conv2d(dim, dim * 3, kernel_size=1, bias=bias)
|
|
||||||
self.qkv_dwconv = nn.Conv2d(
|
|
||||||
dim * 3,
|
|
||||||
dim * 3,
|
|
||||||
kernel_size=3,
|
|
||||||
stride=1,
|
|
||||||
padding=1,
|
|
||||||
groups=dim * 3,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.project_out = nn.Conv2d(dim, dim, kernel_size=1, bias=bias)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
b, c, h, w = x.shape
|
|
||||||
|
|
||||||
qkv = self.qkv_dwconv(self.qkv(x))
|
|
||||||
qkv = qkv.chunk(3, dim=1)
|
|
||||||
|
|
||||||
q, k, v = map(
|
|
||||||
lambda t: rearrange(
|
|
||||||
t,
|
|
||||||
"b (head d) (h ph) (w pw) -> b (ph pw) head d (h w)",
|
|
||||||
ph=self.ps,
|
|
||||||
pw=self.ps,
|
|
||||||
head=self.heads,
|
|
||||||
),
|
|
||||||
qkv,
|
|
||||||
)
|
|
||||||
|
|
||||||
q = F.normalize(q, dim=-1)
|
|
||||||
k = F.normalize(k, dim=-1)
|
|
||||||
|
|
||||||
attn = (q @ k.transpose(-2, -1)) * self.temperature
|
|
||||||
attn = attn.softmax(dim=-1)
|
|
||||||
out = attn @ v
|
|
||||||
|
|
||||||
out = rearrange(
|
|
||||||
out,
|
|
||||||
"b (ph pw) head d (h w) -> b (head d) (h ph) (w pw)",
|
|
||||||
h=h // self.ps,
|
|
||||||
w=w // self.ps,
|
|
||||||
ph=self.ps,
|
|
||||||
pw=self.ps,
|
|
||||||
head=self.heads,
|
|
||||||
)
|
|
||||||
|
|
||||||
out = self.project_out(out)
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
class OSA_Block(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
channel_num=64,
|
|
||||||
bias=True,
|
|
||||||
ffn_bias=True,
|
|
||||||
window_size=8,
|
|
||||||
with_pe=False,
|
|
||||||
dropout=0.0,
|
|
||||||
):
|
|
||||||
super(OSA_Block, self).__init__()
|
|
||||||
|
|
||||||
w = window_size
|
|
||||||
|
|
||||||
self.layer = nn.Sequential(
|
|
||||||
MBConv(
|
|
||||||
channel_num,
|
|
||||||
channel_num,
|
|
||||||
downsample=False,
|
|
||||||
expansion_rate=1,
|
|
||||||
shrinkage_rate=0.25,
|
|
||||||
),
|
|
||||||
Rearrange(
|
|
||||||
"b d (x w1) (y w2) -> b x y w1 w2 d", w1=w, w2=w
|
|
||||||
), # block-like attention
|
|
||||||
PreNormResidual(
|
|
||||||
channel_num,
|
|
||||||
Attention(
|
|
||||||
dim=channel_num,
|
|
||||||
dim_head=channel_num // 4,
|
|
||||||
dropout=dropout,
|
|
||||||
window_size=window_size,
|
|
||||||
with_pe=with_pe,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
Rearrange("b x y w1 w2 d -> b d (x w1) (y w2)"),
|
|
||||||
Conv_PreNormResidual(
|
|
||||||
channel_num, Gated_Conv_FeedForward(dim=channel_num, dropout=dropout)
|
|
||||||
),
|
|
||||||
# channel-like attention
|
|
||||||
Conv_PreNormResidual(
|
|
||||||
channel_num,
|
|
||||||
Channel_Attention(
|
|
||||||
dim=channel_num, heads=4, dropout=dropout, window_size=window_size
|
|
||||||
),
|
|
||||||
),
|
|
||||||
Conv_PreNormResidual(
|
|
||||||
channel_num, Gated_Conv_FeedForward(dim=channel_num, dropout=dropout)
|
|
||||||
),
|
|
||||||
Rearrange(
|
|
||||||
"b d (w1 x) (w2 y) -> b x y w1 w2 d", w1=w, w2=w
|
|
||||||
), # grid-like attention
|
|
||||||
PreNormResidual(
|
|
||||||
channel_num,
|
|
||||||
Attention(
|
|
||||||
dim=channel_num,
|
|
||||||
dim_head=channel_num // 4,
|
|
||||||
dropout=dropout,
|
|
||||||
window_size=window_size,
|
|
||||||
with_pe=with_pe,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
Rearrange("b x y w1 w2 d -> b d (w1 x) (w2 y)"),
|
|
||||||
Conv_PreNormResidual(
|
|
||||||
channel_num, Gated_Conv_FeedForward(dim=channel_num, dropout=dropout)
|
|
||||||
),
|
|
||||||
# channel-like attention
|
|
||||||
Conv_PreNormResidual(
|
|
||||||
channel_num,
|
|
||||||
Channel_Attention_grid(
|
|
||||||
dim=channel_num, heads=4, dropout=dropout, window_size=window_size
|
|
||||||
),
|
|
||||||
),
|
|
||||||
Conv_PreNormResidual(
|
|
||||||
channel_num, Gated_Conv_FeedForward(dim=channel_num, dropout=dropout)
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
out = self.layer(x)
|
|
||||||
return out
|
|
||||||
@ -1,60 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding:utf-8 -*-
|
|
||||||
#############################################################
|
|
||||||
# File: OSAG.py
|
|
||||||
# Created Date: Tuesday April 28th 2022
|
|
||||||
# Author: Chen Xuanhong
|
|
||||||
# Email: chenxuanhongzju@outlook.com
|
|
||||||
# Last Modified: Sunday, 23rd April 2023 3:08:49 pm
|
|
||||||
# Modified By: Chen Xuanhong
|
|
||||||
# Copyright (c) 2020 Shanghai Jiao Tong University
|
|
||||||
#############################################################
|
|
||||||
|
|
||||||
|
|
||||||
import torch.nn as nn
|
|
||||||
|
|
||||||
from .esa import ESA
|
|
||||||
from .OSA import OSA_Block
|
|
||||||
|
|
||||||
|
|
||||||
class OSAG(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
channel_num=64,
|
|
||||||
bias=True,
|
|
||||||
block_num=4,
|
|
||||||
ffn_bias=False,
|
|
||||||
window_size=0,
|
|
||||||
pe=False,
|
|
||||||
):
|
|
||||||
super(OSAG, self).__init__()
|
|
||||||
|
|
||||||
# print("window_size: %d" % (window_size))
|
|
||||||
# print("with_pe", pe)
|
|
||||||
# print("ffn_bias: %d" % (ffn_bias))
|
|
||||||
|
|
||||||
# block_script_name = kwargs.get("block_script_name", "OSA")
|
|
||||||
# block_class_name = kwargs.get("block_class_name", "OSA_Block")
|
|
||||||
|
|
||||||
# script_name = "." + block_script_name
|
|
||||||
# package = __import__(script_name, fromlist=True)
|
|
||||||
block_class = OSA_Block # getattr(package, block_class_name)
|
|
||||||
group_list = []
|
|
||||||
for _ in range(block_num):
|
|
||||||
temp_res = block_class(
|
|
||||||
channel_num,
|
|
||||||
bias,
|
|
||||||
ffn_bias=ffn_bias,
|
|
||||||
window_size=window_size,
|
|
||||||
with_pe=pe,
|
|
||||||
)
|
|
||||||
group_list.append(temp_res)
|
|
||||||
group_list.append(nn.Conv2d(channel_num, channel_num, 1, 1, 0, bias=bias))
|
|
||||||
self.residual_layer = nn.Sequential(*group_list)
|
|
||||||
esa_channel = max(channel_num // 4, 16)
|
|
||||||
self.esa = ESA(esa_channel, channel_num)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
out = self.residual_layer(x)
|
|
||||||
out = out + x
|
|
||||||
return self.esa(out)
|
|
||||||
@ -1,143 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding:utf-8 -*-
|
|
||||||
#############################################################
|
|
||||||
# File: OmniSR.py
|
|
||||||
# Created Date: Tuesday April 28th 2022
|
|
||||||
# Author: Chen Xuanhong
|
|
||||||
# Email: chenxuanhongzju@outlook.com
|
|
||||||
# Last Modified: Sunday, 23rd April 2023 3:06:36 pm
|
|
||||||
# Modified By: Chen Xuanhong
|
|
||||||
# Copyright (c) 2020 Shanghai Jiao Tong University
|
|
||||||
#############################################################
|
|
||||||
|
|
||||||
import math
|
|
||||||
|
|
||||||
import torch
|
|
||||||
import torch.nn as nn
|
|
||||||
import torch.nn.functional as F
|
|
||||||
|
|
||||||
from .OSAG import OSAG
|
|
||||||
from .pixelshuffle import pixelshuffle_block
|
|
||||||
|
|
||||||
|
|
||||||
class OmniSR(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
state_dict,
|
|
||||||
**kwargs,
|
|
||||||
):
|
|
||||||
super(OmniSR, self).__init__()
|
|
||||||
self.state = state_dict
|
|
||||||
|
|
||||||
bias = True # Fine to assume this for now
|
|
||||||
block_num = 1 # Fine to assume this for now
|
|
||||||
ffn_bias = True
|
|
||||||
pe = True
|
|
||||||
|
|
||||||
num_feat = state_dict["input.weight"].shape[0] or 64
|
|
||||||
num_in_ch = state_dict["input.weight"].shape[1] or 3
|
|
||||||
num_out_ch = num_in_ch # we can just assume this for now. pixelshuffle smh
|
|
||||||
|
|
||||||
pixelshuffle_shape = state_dict["up.0.weight"].shape[0]
|
|
||||||
up_scale = math.sqrt(pixelshuffle_shape / num_out_ch)
|
|
||||||
if up_scale - int(up_scale) > 0:
|
|
||||||
print(
|
|
||||||
"out_nc is probably different than in_nc, scale calculation might be wrong"
|
|
||||||
)
|
|
||||||
up_scale = int(up_scale)
|
|
||||||
res_num = 0
|
|
||||||
for key in state_dict.keys():
|
|
||||||
if "residual_layer" in key:
|
|
||||||
temp_res_num = int(key.split(".")[1])
|
|
||||||
if temp_res_num > res_num:
|
|
||||||
res_num = temp_res_num
|
|
||||||
res_num = res_num + 1 # zero-indexed
|
|
||||||
|
|
||||||
residual_layer = []
|
|
||||||
self.res_num = res_num
|
|
||||||
|
|
||||||
if (
|
|
||||||
"residual_layer.0.residual_layer.0.layer.2.fn.rel_pos_bias.weight"
|
|
||||||
in state_dict.keys()
|
|
||||||
):
|
|
||||||
rel_pos_bias_weight = state_dict[
|
|
||||||
"residual_layer.0.residual_layer.0.layer.2.fn.rel_pos_bias.weight"
|
|
||||||
].shape[0]
|
|
||||||
self.window_size = int((math.sqrt(rel_pos_bias_weight) + 1) / 2)
|
|
||||||
else:
|
|
||||||
self.window_size = 8
|
|
||||||
|
|
||||||
self.up_scale = up_scale
|
|
||||||
|
|
||||||
for _ in range(res_num):
|
|
||||||
temp_res = OSAG(
|
|
||||||
channel_num=num_feat,
|
|
||||||
bias=bias,
|
|
||||||
block_num=block_num,
|
|
||||||
ffn_bias=ffn_bias,
|
|
||||||
window_size=self.window_size,
|
|
||||||
pe=pe,
|
|
||||||
)
|
|
||||||
residual_layer.append(temp_res)
|
|
||||||
self.residual_layer = nn.Sequential(*residual_layer)
|
|
||||||
self.input = nn.Conv2d(
|
|
||||||
in_channels=num_in_ch,
|
|
||||||
out_channels=num_feat,
|
|
||||||
kernel_size=3,
|
|
||||||
stride=1,
|
|
||||||
padding=1,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.output = nn.Conv2d(
|
|
||||||
in_channels=num_feat,
|
|
||||||
out_channels=num_feat,
|
|
||||||
kernel_size=3,
|
|
||||||
stride=1,
|
|
||||||
padding=1,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.up = pixelshuffle_block(num_feat, num_out_ch, up_scale, bias=bias)
|
|
||||||
|
|
||||||
# self.tail = pixelshuffle_block(num_feat,num_out_ch,up_scale,bias=bias)
|
|
||||||
|
|
||||||
# for m in self.modules():
|
|
||||||
# if isinstance(m, nn.Conv2d):
|
|
||||||
# n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
|
|
||||||
# m.weight.data.normal_(0, sqrt(2. / n))
|
|
||||||
|
|
||||||
# chaiNNer specific stuff
|
|
||||||
self.model_arch = "OmniSR"
|
|
||||||
self.sub_type = "SR"
|
|
||||||
self.in_nc = num_in_ch
|
|
||||||
self.out_nc = num_out_ch
|
|
||||||
self.num_feat = num_feat
|
|
||||||
self.scale = up_scale
|
|
||||||
|
|
||||||
self.supports_fp16 = True # TODO: Test this
|
|
||||||
self.supports_bfp16 = True
|
|
||||||
self.min_size_restriction = 16
|
|
||||||
|
|
||||||
self.load_state_dict(state_dict, strict=False)
|
|
||||||
|
|
||||||
def check_image_size(self, x):
|
|
||||||
_, _, h, w = x.size()
|
|
||||||
# import pdb; pdb.set_trace()
|
|
||||||
mod_pad_h = (self.window_size - h % self.window_size) % self.window_size
|
|
||||||
mod_pad_w = (self.window_size - w % self.window_size) % self.window_size
|
|
||||||
# x = F.pad(x, (0, mod_pad_w, 0, mod_pad_h), 'reflect')
|
|
||||||
x = F.pad(x, (0, mod_pad_w, 0, mod_pad_h), "constant", 0)
|
|
||||||
return x
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
H, W = x.shape[2:]
|
|
||||||
x = self.check_image_size(x)
|
|
||||||
|
|
||||||
residual = self.input(x)
|
|
||||||
out = self.residual_layer(residual)
|
|
||||||
|
|
||||||
# origin
|
|
||||||
out = torch.add(self.output(out), residual)
|
|
||||||
out = self.up(out)
|
|
||||||
|
|
||||||
out = out[:, :, : H * self.up_scale, : W * self.up_scale]
|
|
||||||
return out
|
|
||||||
@ -1,294 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding:utf-8 -*-
|
|
||||||
#############################################################
|
|
||||||
# File: esa.py
|
|
||||||
# Created Date: Tuesday April 28th 2022
|
|
||||||
# Author: Chen Xuanhong
|
|
||||||
# Email: chenxuanhongzju@outlook.com
|
|
||||||
# Last Modified: Thursday, 20th April 2023 9:28:06 am
|
|
||||||
# Modified By: Chen Xuanhong
|
|
||||||
# Copyright (c) 2020 Shanghai Jiao Tong University
|
|
||||||
#############################################################
|
|
||||||
|
|
||||||
import torch
|
|
||||||
import torch.nn as nn
|
|
||||||
import torch.nn.functional as F
|
|
||||||
|
|
||||||
from .layernorm import LayerNorm2d
|
|
||||||
|
|
||||||
|
|
||||||
def moment(x, dim=(2, 3), k=2):
|
|
||||||
assert len(x.size()) == 4
|
|
||||||
mean = torch.mean(x, dim=dim).unsqueeze(-1).unsqueeze(-1)
|
|
||||||
mk = (1 / (x.size(2) * x.size(3))) * torch.sum(torch.pow(x - mean, k), dim=dim)
|
|
||||||
return mk
|
|
||||||
|
|
||||||
|
|
||||||
class ESA(nn.Module):
|
|
||||||
"""
|
|
||||||
Modification of Enhanced Spatial Attention (ESA), which is proposed by
|
|
||||||
`Residual Feature Aggregation Network for Image Super-Resolution`
|
|
||||||
Note: `conv_max` and `conv3_` are NOT used here, so the corresponding codes
|
|
||||||
are deleted.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, esa_channels, n_feats, conv=nn.Conv2d):
|
|
||||||
super(ESA, self).__init__()
|
|
||||||
f = esa_channels
|
|
||||||
self.conv1 = conv(n_feats, f, kernel_size=1)
|
|
||||||
self.conv_f = conv(f, f, kernel_size=1)
|
|
||||||
self.conv2 = conv(f, f, kernel_size=3, stride=2, padding=0)
|
|
||||||
self.conv3 = conv(f, f, kernel_size=3, padding=1)
|
|
||||||
self.conv4 = conv(f, n_feats, kernel_size=1)
|
|
||||||
self.sigmoid = nn.Sigmoid()
|
|
||||||
self.relu = nn.ReLU(inplace=True)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
c1_ = self.conv1(x)
|
|
||||||
c1 = self.conv2(c1_)
|
|
||||||
v_max = F.max_pool2d(c1, kernel_size=7, stride=3)
|
|
||||||
c3 = self.conv3(v_max)
|
|
||||||
c3 = F.interpolate(
|
|
||||||
c3, (x.size(2), x.size(3)), mode="bilinear", align_corners=False
|
|
||||||
)
|
|
||||||
cf = self.conv_f(c1_)
|
|
||||||
c4 = self.conv4(c3 + cf)
|
|
||||||
m = self.sigmoid(c4)
|
|
||||||
return x * m
|
|
||||||
|
|
||||||
|
|
||||||
class LK_ESA(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self, esa_channels, n_feats, conv=nn.Conv2d, kernel_expand=1, bias=True
|
|
||||||
):
|
|
||||||
super(LK_ESA, self).__init__()
|
|
||||||
f = esa_channels
|
|
||||||
self.conv1 = conv(n_feats, f, kernel_size=1)
|
|
||||||
self.conv_f = conv(f, f, kernel_size=1)
|
|
||||||
|
|
||||||
kernel_size = 17
|
|
||||||
kernel_expand = kernel_expand
|
|
||||||
padding = kernel_size // 2
|
|
||||||
|
|
||||||
self.vec_conv = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(1, kernel_size),
|
|
||||||
padding=(0, padding),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.vec_conv3x1 = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(1, 3),
|
|
||||||
padding=(0, 1),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.hor_conv = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(kernel_size, 1),
|
|
||||||
padding=(padding, 0),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.hor_conv1x3 = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(3, 1),
|
|
||||||
padding=(1, 0),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.conv4 = conv(f, n_feats, kernel_size=1)
|
|
||||||
self.sigmoid = nn.Sigmoid()
|
|
||||||
self.relu = nn.ReLU(inplace=True)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
c1_ = self.conv1(x)
|
|
||||||
|
|
||||||
res = self.vec_conv(c1_) + self.vec_conv3x1(c1_)
|
|
||||||
res = self.hor_conv(res) + self.hor_conv1x3(res)
|
|
||||||
|
|
||||||
cf = self.conv_f(c1_)
|
|
||||||
c4 = self.conv4(res + cf)
|
|
||||||
m = self.sigmoid(c4)
|
|
||||||
return x * m
|
|
||||||
|
|
||||||
|
|
||||||
class LK_ESA_LN(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self, esa_channels, n_feats, conv=nn.Conv2d, kernel_expand=1, bias=True
|
|
||||||
):
|
|
||||||
super(LK_ESA_LN, self).__init__()
|
|
||||||
f = esa_channels
|
|
||||||
self.conv1 = conv(n_feats, f, kernel_size=1)
|
|
||||||
self.conv_f = conv(f, f, kernel_size=1)
|
|
||||||
|
|
||||||
kernel_size = 17
|
|
||||||
kernel_expand = kernel_expand
|
|
||||||
padding = kernel_size // 2
|
|
||||||
|
|
||||||
self.norm = LayerNorm2d(n_feats)
|
|
||||||
|
|
||||||
self.vec_conv = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(1, kernel_size),
|
|
||||||
padding=(0, padding),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.vec_conv3x1 = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(1, 3),
|
|
||||||
padding=(0, 1),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.hor_conv = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(kernel_size, 1),
|
|
||||||
padding=(padding, 0),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
self.hor_conv1x3 = nn.Conv2d(
|
|
||||||
in_channels=f * kernel_expand,
|
|
||||||
out_channels=f * kernel_expand,
|
|
||||||
kernel_size=(3, 1),
|
|
||||||
padding=(1, 0),
|
|
||||||
groups=2,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.conv4 = conv(f, n_feats, kernel_size=1)
|
|
||||||
self.sigmoid = nn.Sigmoid()
|
|
||||||
self.relu = nn.ReLU(inplace=True)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
c1_ = self.norm(x)
|
|
||||||
c1_ = self.conv1(c1_)
|
|
||||||
|
|
||||||
res = self.vec_conv(c1_) + self.vec_conv3x1(c1_)
|
|
||||||
res = self.hor_conv(res) + self.hor_conv1x3(res)
|
|
||||||
|
|
||||||
cf = self.conv_f(c1_)
|
|
||||||
c4 = self.conv4(res + cf)
|
|
||||||
m = self.sigmoid(c4)
|
|
||||||
return x * m
|
|
||||||
|
|
||||||
|
|
||||||
class AdaGuidedFilter(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self, esa_channels, n_feats, conv=nn.Conv2d, kernel_expand=1, bias=True
|
|
||||||
):
|
|
||||||
super(AdaGuidedFilter, self).__init__()
|
|
||||||
|
|
||||||
self.gap = nn.AdaptiveAvgPool2d(1)
|
|
||||||
self.fc = nn.Conv2d(
|
|
||||||
in_channels=n_feats,
|
|
||||||
out_channels=1,
|
|
||||||
kernel_size=1,
|
|
||||||
padding=0,
|
|
||||||
stride=1,
|
|
||||||
groups=1,
|
|
||||||
bias=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.r = 5
|
|
||||||
|
|
||||||
def box_filter(self, x, r):
|
|
||||||
channel = x.shape[1]
|
|
||||||
kernel_size = 2 * r + 1
|
|
||||||
weight = 1.0 / (kernel_size**2)
|
|
||||||
box_kernel = weight * torch.ones(
|
|
||||||
(channel, 1, kernel_size, kernel_size), dtype=torch.float32, device=x.device
|
|
||||||
)
|
|
||||||
output = F.conv2d(x, weight=box_kernel, stride=1, padding=r, groups=channel)
|
|
||||||
return output
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
_, _, H, W = x.shape
|
|
||||||
N = self.box_filter(
|
|
||||||
torch.ones((1, 1, H, W), dtype=x.dtype, device=x.device), self.r
|
|
||||||
)
|
|
||||||
|
|
||||||
# epsilon = self.fc(self.gap(x))
|
|
||||||
# epsilon = torch.pow(epsilon, 2)
|
|
||||||
epsilon = 1e-2
|
|
||||||
|
|
||||||
mean_x = self.box_filter(x, self.r) / N
|
|
||||||
var_x = self.box_filter(x * x, self.r) / N - mean_x * mean_x
|
|
||||||
|
|
||||||
A = var_x / (var_x + epsilon)
|
|
||||||
b = (1 - A) * mean_x
|
|
||||||
m = A * x + b
|
|
||||||
|
|
||||||
# mean_A = self.box_filter(A, self.r) / N
|
|
||||||
# mean_b = self.box_filter(b, self.r) / N
|
|
||||||
# m = mean_A * x + mean_b
|
|
||||||
return x * m
|
|
||||||
|
|
||||||
|
|
||||||
class AdaConvGuidedFilter(nn.Module):
|
|
||||||
def __init__(
|
|
||||||
self, esa_channels, n_feats, conv=nn.Conv2d, kernel_expand=1, bias=True
|
|
||||||
):
|
|
||||||
super(AdaConvGuidedFilter, self).__init__()
|
|
||||||
f = esa_channels
|
|
||||||
|
|
||||||
self.conv_f = conv(f, f, kernel_size=1)
|
|
||||||
|
|
||||||
kernel_size = 17
|
|
||||||
kernel_expand = kernel_expand
|
|
||||||
padding = kernel_size // 2
|
|
||||||
|
|
||||||
self.vec_conv = nn.Conv2d(
|
|
||||||
in_channels=f,
|
|
||||||
out_channels=f,
|
|
||||||
kernel_size=(1, kernel_size),
|
|
||||||
padding=(0, padding),
|
|
||||||
groups=f,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.hor_conv = nn.Conv2d(
|
|
||||||
in_channels=f,
|
|
||||||
out_channels=f,
|
|
||||||
kernel_size=(kernel_size, 1),
|
|
||||||
padding=(padding, 0),
|
|
||||||
groups=f,
|
|
||||||
bias=bias,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.gap = nn.AdaptiveAvgPool2d(1)
|
|
||||||
self.fc = nn.Conv2d(
|
|
||||||
in_channels=f,
|
|
||||||
out_channels=f,
|
|
||||||
kernel_size=1,
|
|
||||||
padding=0,
|
|
||||||
stride=1,
|
|
||||||
groups=1,
|
|
||||||
bias=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
y = self.vec_conv(x)
|
|
||||||
y = self.hor_conv(y)
|
|
||||||
|
|
||||||
sigma = torch.pow(y, 2)
|
|
||||||
epsilon = self.fc(self.gap(y))
|
|
||||||
|
|
||||||
weight = sigma / (sigma + epsilon)
|
|
||||||
|
|
||||||
m = weight * x + (1 - weight)
|
|
||||||
|
|
||||||
return x * m
|
|
||||||
@ -1,70 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding:utf-8 -*-
|
|
||||||
#############################################################
|
|
||||||
# File: layernorm.py
|
|
||||||
# Created Date: Tuesday April 28th 2022
|
|
||||||
# Author: Chen Xuanhong
|
|
||||||
# Email: chenxuanhongzju@outlook.com
|
|
||||||
# Last Modified: Thursday, 20th April 2023 9:28:20 am
|
|
||||||
# Modified By: Chen Xuanhong
|
|
||||||
# Copyright (c) 2020 Shanghai Jiao Tong University
|
|
||||||
#############################################################
|
|
||||||
|
|
||||||
import torch
|
|
||||||
import torch.nn as nn
|
|
||||||
|
|
||||||
|
|
||||||
class LayerNormFunction(torch.autograd.Function):
|
|
||||||
@staticmethod
|
|
||||||
def forward(ctx, x, weight, bias, eps):
|
|
||||||
ctx.eps = eps
|
|
||||||
N, C, H, W = x.size()
|
|
||||||
mu = x.mean(1, keepdim=True)
|
|
||||||
var = (x - mu).pow(2).mean(1, keepdim=True)
|
|
||||||
y = (x - mu) / (var + eps).sqrt()
|
|
||||||
ctx.save_for_backward(y, var, weight)
|
|
||||||
y = weight.view(1, C, 1, 1) * y + bias.view(1, C, 1, 1)
|
|
||||||
return y
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def backward(ctx, grad_output):
|
|
||||||
eps = ctx.eps
|
|
||||||
|
|
||||||
N, C, H, W = grad_output.size()
|
|
||||||
y, var, weight = ctx.saved_variables
|
|
||||||
g = grad_output * weight.view(1, C, 1, 1)
|
|
||||||
mean_g = g.mean(dim=1, keepdim=True)
|
|
||||||
|
|
||||||
mean_gy = (g * y).mean(dim=1, keepdim=True)
|
|
||||||
gx = 1.0 / torch.sqrt(var + eps) * (g - y * mean_gy - mean_g)
|
|
||||||
return (
|
|
||||||
gx,
|
|
||||||
(grad_output * y).sum(dim=3).sum(dim=2).sum(dim=0),
|
|
||||||
grad_output.sum(dim=3).sum(dim=2).sum(dim=0),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class LayerNorm2d(nn.Module):
|
|
||||||
def __init__(self, channels, eps=1e-6):
|
|
||||||
super(LayerNorm2d, self).__init__()
|
|
||||||
self.register_parameter("weight", nn.Parameter(torch.ones(channels)))
|
|
||||||
self.register_parameter("bias", nn.Parameter(torch.zeros(channels)))
|
|
||||||
self.eps = eps
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return LayerNormFunction.apply(x, self.weight, self.bias, self.eps)
|
|
||||||
|
|
||||||
|
|
||||||
class GRN(nn.Module):
|
|
||||||
"""GRN (Global Response Normalization) layer"""
|
|
||||||
|
|
||||||
def __init__(self, dim):
|
|
||||||
super().__init__()
|
|
||||||
self.gamma = nn.Parameter(torch.zeros(1, dim, 1, 1))
|
|
||||||
self.beta = nn.Parameter(torch.zeros(1, dim, 1, 1))
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
Gx = torch.norm(x, p=2, dim=(2, 3), keepdim=True)
|
|
||||||
Nx = Gx / (Gx.mean(dim=1, keepdim=True) + 1e-6)
|
|
||||||
return self.gamma * (x * Nx) + self.beta + x
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user