Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: extract_to_frame can not be higher than extract_from_frame #882

Open
3 tasks done
GeorgeSpiller opened this issue Sep 14, 2023 · 5 comments
Open
3 tasks done
Labels
bug Something isn't working

Comments

@GeorgeSpiller
Copy link

Have you read the latest version of the FAQ?

  • I have visited the FAQ page right now and my issue is not present there

Is there an existing issue for this?

  • I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Are you using the latest version of the Deforum extension?

  • I have Deforum updated to the lastest version and I still have the issue.

What happened?

Exception thrown when initially rendering with only contorl nets:

webui/scripts/deforum_helpers/video_audio_utilities.py", line 65, in vid2frames
raise RuntimeError('Error: extract_to_frame can not be higher than extract_from_frame')
RuntimeError: Error: extract_to_frame can not be higher than extract_from_frame

Steps to reproduce the problem

Set Hybrid Video -> Composite and Motion to None
Have at least one ControlNet, with the correct video path
Set Keyframes -> Max Frames to -1

As far as I can tell, all other settings can be changed (including init -> extract_to and extract_from, which can both be a number or for extract to -1). It should yield something akin to the attached in the CLI.
cli_out.txt

What should have happened/how would you fix it?

All the frames for the init video should have been extracted to start off the Diffusion process.

I think this error is a result of the Keyframes -> Max Frames set to -1, as if this is a positive int it works fine. Also switching Hybrid Video -> Composite -> Normal and having Keyframes -> Max Frames -1 results in no issues. (Composite has to be None and max frames -1 for the issue to happen) Hope this helps!

Torch version

python: 3.10.12
torch: 2.0.0+cu118

On which platform are you launching the webui with the extension?

Google Colab (Other)

Deforum settings

20230914101540_settings.txt

Webui core settings

ui-config.zip
This box does not accept .json files, so Its attached within a .zip.

Console logs

/content
Mounted at /content/drive
env: PYTHONDONTWRITEBYTECODE=1
env: TF_CPP_MIN_LOG_LEVEL=1
28 packages can be upgraded. Run 'apt list --upgradable' to see them.
--2023-09-14 13:13:33--  https://github.com/camenduru/gperftools/releases/download/v1.0/libtcmalloc_minimal.so.4
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/669786276/620e2e64-be9f-4599-904f-18ee3811e159?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230914%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230914T131333Z&X-Amz-Expires=300&X-Amz-Signature=6508c5da845f11d3ce1b406b2814553448122d7384bf23b03e2d6617e032531a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=669786276&response-content-disposition=attachment%3B%20filename%3Dlibtcmalloc_minimal.so.4&response-content-type=application%2Foctet-stream [following]
--2023-09-14 13:13:33--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/669786276/620e2e64-be9f-4599-904f-18ee3811e159?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230914%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230914T131333Z&X-Amz-Expires=300&X-Amz-Signature=6508c5da845f11d3ce1b406b2814553448122d7384bf23b03e2d6617e032531a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=669786276&response-content-disposition=attachment%3B%20filename%3Dlibtcmalloc_minimal.so.4&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 373960 (365K) [application/octet-stream]
Saving to: ‘/content/libtcmalloc_minimal.so.4’

/content/libtcmallo 100%[===================>] 365.20K  --.-KB/s    in 0.005s  

2023-09-14 13:13:33 (67.8 MB/s) - ‘/content/libtcmalloc_minimal.so.4’ saved [373960/373960]

env: LD_PRELOAD=/content/libtcmalloc_minimal.so.4
pkg-config is already the newest version (0.29.2-1ubuntu3).
python3-dev is already the newest version (3.10.6-1~22.04).
python3-dev set to manually installed.
The following additional packages will be installed:
  libaria2-0 libblkid-dev libc-ares2 libcairo-script-interpreter2 libffi-dev
  libglib2.0-dev libglib2.0-dev-bin libice-dev liblzo2-2 libmount-dev
  libpixman-1-dev libselinux1-dev libsepol-dev libsm-dev libxcb-render0-dev
  libxcb-shm0-dev
Suggested packages:
  libcairo2-doc libgirepository1.0-dev libglib2.0-doc libgdk-pixbuf2.0-bin
  | libgdk-pixbuf2.0-dev libxml2-utils libice-doc libsm-doc
The following NEW packages will be installed:
  aria2 libaria2-0 libblkid-dev libc-ares2 libcairo-script-interpreter2
  libcairo2-dev libffi-dev libglib2.0-dev libglib2.0-dev-bin libice-dev
  liblzo2-2 libmount-dev libpixman-1-dev libselinux1-dev libsepol-dev
  libsm-dev libxcb-render0-dev libxcb-shm0-dev
0 upgraded, 18 newly installed, 0 to remove and 28 not upgraded.
Need to get 5,352 kB of archives.
After this operation, 25.2 MB of additional disk space will be used.
Selecting previously unselected package libc-ares2:amd64.
(Reading database ... 120901 files and directories currently installed.)
Preparing to unpack .../00-libc-ares2_1.18.1-1ubuntu0.22.04.2_amd64.deb ...
Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...
Selecting previously unselected package libaria2-0:amd64.
Preparing to unpack .../01-libaria2-0_1.36.0-1_amd64.deb ...
Unpacking libaria2-0:amd64 (1.36.0-1) ...
Selecting previously unselected package aria2.
Preparing to unpack .../02-aria2_1.36.0-1_amd64.deb ...
Unpacking aria2 (1.36.0-1) ...
Selecting previously unselected package liblzo2-2:amd64.
Preparing to unpack .../03-liblzo2-2_2.10-2build3_amd64.deb ...
Unpacking liblzo2-2:amd64 (2.10-2build3) ...
Selecting previously unselected package libcairo-script-interpreter2:amd64.
Preparing to unpack .../04-libcairo-script-interpreter2_1.16.0-5ubuntu2_amd64.deb ...
Unpacking libcairo-script-interpreter2:amd64 (1.16.0-5ubuntu2) ...
Selecting previously unselected package libice-dev:amd64.
Preparing to unpack .../05-libice-dev_2%3a1.0.10-1build2_amd64.deb ...
Unpacking libice-dev:amd64 (2:1.0.10-1build2) ...
Selecting previously unselected package libsm-dev:amd64.
Preparing to unpack .../06-libsm-dev_2%3a1.2.3-1build2_amd64.deb ...
Unpacking libsm-dev:amd64 (2:1.2.3-1build2) ...
Selecting previously unselected package libpixman-1-dev:amd64.
Preparing to unpack .../07-libpixman-1-dev_0.40.0-1ubuntu0.22.04.1_amd64.deb ...
Unpacking libpixman-1-dev:amd64 (0.40.0-1ubuntu0.22.04.1) ...
Selecting previously unselected package libxcb-render0-dev:amd64.
Preparing to unpack .../08-libxcb-render0-dev_1.14-3ubuntu3_amd64.deb ...
Unpacking libxcb-render0-dev:amd64 (1.14-3ubuntu3) ...
Selecting previously unselected package libxcb-shm0-dev:amd64.
Preparing to unpack .../09-libxcb-shm0-dev_1.14-3ubuntu3_amd64.deb ...
Unpacking libxcb-shm0-dev:amd64 (1.14-3ubuntu3) ...
Selecting previously unselected package libffi-dev:amd64.
Preparing to unpack .../10-libffi-dev_3.4.2-4_amd64.deb ...
Unpacking libffi-dev:amd64 (3.4.2-4) ...
Selecting previously unselected package libglib2.0-dev-bin.
Preparing to unpack .../11-libglib2.0-dev-bin_2.72.4-0ubuntu2.2_amd64.deb ...
Unpacking libglib2.0-dev-bin (2.72.4-0ubuntu2.2) ...
Selecting previously unselected package libblkid-dev:amd64.
Preparing to unpack .../12-libblkid-dev_2.37.2-4ubuntu3_amd64.deb ...
Unpacking libblkid-dev:amd64 (2.37.2-4ubuntu3) ...
Selecting previously unselected package libsepol-dev:amd64.
Preparing to unpack .../13-libsepol-dev_3.3-1build1_amd64.deb ...
Unpacking libsepol-dev:amd64 (3.3-1build1) ...
Selecting previously unselected package libselinux1-dev:amd64.
Preparing to unpack .../14-libselinux1-dev_3.3-1build2_amd64.deb ...
Unpacking libselinux1-dev:amd64 (3.3-1build2) ...
Selecting previously unselected package libmount-dev:amd64.
Preparing to unpack .../15-libmount-dev_2.37.2-4ubuntu3_amd64.deb ...
Unpacking libmount-dev:amd64 (2.37.2-4ubuntu3) ...
Selecting previously unselected package libglib2.0-dev:amd64.
Preparing to unpack .../16-libglib2.0-dev_2.72.4-0ubuntu2.2_amd64.deb ...
Unpacking libglib2.0-dev:amd64 (2.72.4-0ubuntu2.2) ...
Selecting previously unselected package libcairo2-dev:amd64.
Preparing to unpack .../17-libcairo2-dev_1.16.0-5ubuntu2_amd64.deb ...
Unpacking libcairo2-dev:amd64 (1.16.0-5ubuntu2) ...
Setting up libglib2.0-dev-bin (2.72.4-0ubuntu2.2) ...
Setting up libblkid-dev:amd64 (2.37.2-4ubuntu3) ...
Setting up libpixman-1-dev:amd64 (0.40.0-1ubuntu0.22.04.1) ...
Setting up libice-dev:amd64 (2:1.0.10-1build2) ...
Setting up libsm-dev:amd64 (2:1.2.3-1build2) ...
Setting up liblzo2-2:amd64 (2.10-2build3) ...
Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...
Setting up libffi-dev:amd64 (3.4.2-4) ...
Setting up libxcb-shm0-dev:amd64 (1.14-3ubuntu3) ...
Setting up libsepol-dev:amd64 (3.3-1build1) ...
Setting up libxcb-render0-dev:amd64 (1.14-3ubuntu3) ...
Setting up libcairo-script-interpreter2:amd64 (1.16.0-5ubuntu2) ...
Setting up libaria2-0:amd64 (1.36.0-1) ...
Setting up libselinux1-dev:amd64 (3.3-1build2) ...
Setting up aria2 (1.36.0-1) ...
Setting up libmount-dev:amd64 (2.37.2-4ubuntu3) ...
Setting up libglib2.0-dev:amd64 (2.72.4-0ubuntu2.2) ...
Processing triggers for libglib2.0-0:amd64 (2.72.4-0ubuntu2.2) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link

Processing triggers for man-db (2.10.2-1) ...
Setting up libcairo2-dev:amd64 (1.16.0-5ubuntu2) ...
The following NEW packages will be installed:
  unionfs-fuse
0 upgraded, 1 newly installed, 0 to remove and 28 not upgraded.
Need to get 48.7 kB of archives.
After this operation, 146 kB of additional disk space will be used.
Selecting previously unselected package unionfs-fuse.
(Reading database ... 121748 files and directories currently installed.)
Preparing to unpack .../unionfs-fuse_1.0-1ubuntu2_amd64.deb ...
Unpacking unionfs-fuse (1.0-1ubuntu2) ...
Setting up unionfs-fuse (1.0-1ubuntu2) ...
Processing triggers for man-db (2.10.2-1) ...
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 2.3/2.3 GB 185.4 MB/s eta 0:00:01tcmalloc: large alloc 2267275264 bytes == 0x5c78810a8000 @ 
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 2.3/2.3 GB 168.9 MB/s eta 0:00:01tcmalloc: large alloc 2834096128 bytes == 0x5c79082e6000 @ 
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 2.3/2.3 GB 168.9 MB/s eta 0:00:01tcmalloc: large alloc 2267275264 bytes == 0x5c78810a8000 @ 
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 GB 890.0 kB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/6.1 MB 82.8 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 59.3 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 47.1 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 81.2 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.2/108.2 MB 14.1 MB/s eta 0:00:00
/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui
Updating files: 100% (203/203), done.
HEAD is now at 22bcc7be attempted fix for infinite loading for settings that some people experience
Updating files: 100% (135/135), done.
HEAD is now at cf1d67a Update modelcard.md
HEAD is now at c202932 GRADIO_TUNNEL online
Already up to date.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d3460e|OK  |       0B/s|/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/models/CLIP/ViT-L-14.pt

Status Legend:
(OK):download completed.
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Commit hash: 22bcc7be428c94e9408f589966c2040187245d81
Installing gfpgan
Installing clip
Installing open_clip
Installing requirements for CodeFormer
Installing requirements for Web UI
Installing Deforum requirement: av
Installing Deforum requirement: pims

Installing sd-webui-controlnet requirement: mediapipe
Installing sd-webui-controlnet requirement: svglib
Installing sd-webui-controlnet requirement: fvcore

Installing pycloudflared

Launching Web UI with arguments: --listen --xformers --enable-insecure-extension-access --theme dark --gradio-queue --clip-models-path /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/models/CLIP --ckpt-dir /content/fused-models --lora-dir /content/fused-lora --multiple
2023-09-14 13:20:00.730518: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/usr/local/lib/python3.10/dist-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
Additional Network extension not installed, Only hijack built-in lora
LoCon Extension hijack built-in lora successfully
2023-09-14 13:22:10,324 - ControlNet - INFO - ControlNet v1.1.234
ControlNet preprocessor location: /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/downloads
2023-09-14 13:22:27,411 - ControlNet - INFO - ControlNet v1.1.234
all detected, remote.moe trying to connect...
Warning: Permanently added 'localhost.run' (RSA) to the list of known hosts.
all detected, cloudflared trying to connect...
Download cloudflared...: 100% 34.8M/34.8M [00:00<00:00, 507MB/s]
Loading weights [4199bcdd14] from /content/fused-models/revAnimated_v122.safetensors
Creating model from config: /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Downloading (…)olve/main/vocab.json: 100% 961k/961k [00:00<00:00, 4.13MB/s]
Downloading (…)olve/main/merges.txt: 100% 525k/525k [00:00<00:00, 52.3MB/s]
Downloading (…)cial_tokens_map.json: 100% 389/389 [00:00<00:00, 2.32MB/s]
Downloading (…)okenizer_config.json: 100% 905/905 [00:00<00:00, 5.83MB/s]
Downloading (…)lve/main/config.json: 100% 4.52k/4.52k [00:00<00:00, 22.7MB/s]
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(10): badhandv4, verybadimagenegative_v1.3, bad_prompt_version2, EasyNegative, bad-image-v2-39000, EasyNegativeV2, bad-hands-5, bad-artist, bad-artist-anime, ng_deepnegative_v1_75t
Model loaded in 154.0s (load weights from disk: 85.8s, create model: 5.1s, apply weights to model: 34.4s, load textual inversion embeddings: 28.6s).
*Deforum ControlNet support: enabled*
Public WebUI Colab URL: None 
Public WebUI Colab URL: https://2ba806d6-11bf-443f.gradio.live 
Public WebUI Colab URL: https://1addd794d95072.lhr.life
Public WebUI Colab URL: https://flooring-where-severe-parliament.trycloudflare.com
Startup time: 495.8s (import torch: 5.2s, import gradio: 1.2s, import ldm: 1.9s, other imports: 13.4s, list SD models: 3.9s, setup codeformer: 20.5s, load scripts: 123.7s, load SD checkpoint: 154.1s, create ui: 171.7s, gradio launch: 0.2s).
started run_deforum
times_to_run: 1
Deforum extension for auto1111 webui
Git commit: ff0680bb (Wed Aug  2 14:00:01 2023)
Starting job task(mobhyct6cd3ry4a)-0...
Unpacking ControlNet 1 base video
Exporting Video Frames to /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/outputs/img2img-images/Deforum_20230914133053/controlnet_1_inputframes...

*START OF TRACEBACK*
Traceback (most recent call last):
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/run_deforum.py", line 94, in run_deforum
    render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/render.py", line 65, in render_animation
    unpack_controlnet_vids(args, anim_args, controlnet_args)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/deforum_controlnet.py", line 321, in unpack_controlnet_vids
    process_controlnet_input_frames(args, anim_args, controlnet_args, vid_path, None, 'inputframes', i)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/deforum_controlnet.py", line 297, in process_controlnet_input_frames
    vid2frames(
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/video_audio_utilities.py", line 65, in vid2frames
    raise RuntimeError('Error: extract_to_frame can not be higher than extract_from_frame')
RuntimeError: Error: extract_to_frame can not be higher than extract_from_frame
*END OF TRACEBACK*

User friendly error message:
Error: Error: extract_to_frame can not be higher than extract_from_frame. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \.

Additional information

Run using the following git repo:
https://github.com/camenduru/stable-diffusion-webui-colab/tree/drive

@GeorgeSpiller GeorgeSpiller added the bug Something isn't working label Sep 14, 2023
@github-actions
Copy link

This issue has been closed due to incorrect formatting. Please address the following mistakes and reopen the issue (click on the 'Reopen' button below):

  • Make sure the issue title has at least 3 words.

@GeorgeSpiller GeorgeSpiller changed the title [Bug]: [Bug]: extract_to_frame can not be higher than extract_from_frame Sep 14, 2023
@andyxr
Copy link
Contributor

andyxr commented Sep 14, 2023

I’ve reopened this now it has a title

@andyxr andyxr reopened this Sep 14, 2023
@github-actions github-actions bot added bug Something isn't working and removed bug Something isn't working wrong format labels Sep 14, 2023
@github-actions
Copy link

Thanks for addressing your formatting mistakes. The issue has been reopened now.

@andyxr
Copy link
Contributor

andyxr commented Sep 14, 2023

@GeorgeSpiller And did you have Generate input frames switched ON in HybridVideo? That’s a must-do if you’re wanting to use any of the HybridVideo features (if you haven’t already run it once with Generate input frames ON)

@reallybigname
Copy link
Member

This is obviously just a sloppy error message. The logic was correct. The error message was wrong. It said:

if (extract_to_frame <= extract_from_frame) and extract_to_frame != -1:
    raise RuntimeError('Error: extract_to_frame cannot be higher than the extract_from_frame')

The original programmer probably meant to say this:

if (extract_to_frame <= extract_from_frame) and extract_to_frame != -1:
    raise RuntimeError('Error: extract_from_frame cannot be higher than the extract_to_frame')

They just swapped the to and from in the message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants