ComfyUI-MuseTalk部署依赖mmcv

ComfyUI-MuseTalk部署依赖mmcv

** ComfyUI-MuseTalk是MuseTalk基于ComfyUI的自定义节点插件。MMPose 是一款基于 PyTorch 的“人体姿态”分析的开源工具箱，是 OpenMMLab 项目的成员之一。OpenMMLab 团队致力于构建了深度学习时代最具影响力的开源计算机视觉算法系统，于 2022 年 9 月 1 日在世界人工智能大会发布了新一代训练引擎 MMEngine，它是一个用于训练深度学习模型的基础库（MMEngine 是一个基于 PyTorch 实现的，用于训练深度学习模型的基础库。它作为 OpenMMLab 所有代码库的训练引擎，其在不同研究领域支持了上百个算法。此外，MMEngine 也可以用于非 OpenMMLab 项目中）。相比于 MMCV**，它提供了更高级且通用的训练器、接口更加统一的开放架构以及可定制化程度更高的训练流程。MMCV v2.0.0 正式版本于 2023 年 4 月 6 日发布。在 2.x 版本中，它删除了和训练流程相关的组件，并新增了数据变换模块。

AIGC文生图音频合成ComfyUI-MuseTalk测试

** **ComfyUI-MuseTalk运行时，默认安装依赖mmpose必须依赖mmcv才能正确执行。

一、现象-运行时错误

二、解决

2.1、如果当前python独立环境配置了国内镜像就不要开墙

   否则http从国外“请求”及“回调”时会出现问题：

2.2、用一键包安装

    当前环境：

    开始“重新安装”：

    仍然有错误，因为它内部直接将其升级安装为mmcv包的当前最新版本2.2.0（截止2024-07-25）：

2.3、手动安装

C:>E:

E:>cd E:\ComfyUI-aki\ComfyUI-aki-v1.3\python

E:\ComfyUI-aki\ComfyUI-aki-v1.3\python>python.exe -m pip show mmcv

E:\ComfyUI-aki\ComfyUI-aki-v1.3\python>python.exe -m pip install mmcv==2.0.0rc4

Microsoft Windows [版本 10.0.19045.4291]
(c) Microsoft Corporation。保留所有权利。

E:\ComfyUI-aki\ComfyUI-aki-v1.3\python>python.exe
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

help> modules mmcv

Here is a list of modules whose name or summary contains 'mmcv'.
If there are any, enter a module name to get more help.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\_sitebuiltins.py", line 103, in __call__
    return pydoc.help(*args, **kwds)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pydoc.py", line 2009, in __call__
    self.interact()
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pydoc.py", line 2036, in interact
    self.help(request)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pydoc.py", line 2055, in help
    self.listmodules(request.split()[1])
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pydoc.py", line 2191, in listmodules
    apropos(key)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pydoc.py", line 2285, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pydoc.py", line 2234, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\pkgutil.py", line 92, in walk_packages
    __import__(info.name)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\clip_interrogator\__init__.py", line 1, in <module>
    from .clip_interrogator import Config, Interrogator, LabelTable, list_caption_models, list_clip_models, load_list
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\clip_interrogator\clip_interrogator.py", line 4, in <module>
    import open_clip
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\open_clip\__init__.py", line 1, in <module>
    from .coca_model import CoCa
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\open_clip\coca_model.py", line 15, in <module>
    from .model import CLIPTextCfg, CLIPVisionCfg, _build_vision_tower, _build_text_tower
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\open_clip\model.py", line 16, in <module>
    from .hf_model import HFTextEncoder
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\open_clip\hf_model.py", line 13, in <module>
    from transformers import AutoModel, AutoTokenizer, AutoConfig, PretrainedConfig
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\transformers\utils\import_utils.py", line 1174, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\transformers\utils\import_utils.py", line 1184, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "E:\ComfyUI-aki\ComfyUI-aki-v1.3\python\lib\site-packages\transformers\models\__init__.py", line 15, in <module>
    from . import (
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1029, in get_code
  File "<frozen importlib._bootstrap_external>", line 1097, in _cache_bytecode
  File "<frozen importlib._bootstrap_external>", line 1111, in set_data

>>> quit()

E:\ComfyUI-aki\ComfyUI-aki-v1.3\python>python.exe -m pip show mmcv
Name: mmcv
Version: 2.2.0
Summary: OpenMMLab Computer Vision Foundation
Home-page: https://github.com/open-mmlab/mmcv
Author: MMCV Contributors
Author-email: [email protected]
License:
Location: e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages
Requires: addict, mmengine, numpy, packaging, Pillow, pyyaml, regex, yapf
Required-by:

E:\ComfyUI-aki\ComfyUI-aki-v1.3\python>python.exe -m pip install mmcv==2.0.0rc4
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting mmcv==2.0.0rc4
  Downloading mmcv-2.0.0rc4.tar.gz (436 kB)
     ---------------------------------------- 436.4/436.4 kB 16.0 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: addict in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (2.4.0)
Requirement already satisfied: mmengine>=0.2.0 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (0.10.4)
Requirement already satisfied: numpy in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (1.26.4)
Requirement already satisfied: packaging in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (24.1)
Requirement already satisfied: Pillow in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (10.4.0)
Requirement already satisfied: pyyaml in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (6.0.1)
Requirement already satisfied: yapf in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (0.40.2)
Requirement already satisfied: regex in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmcv==2.0.0rc4) (2024.7.24)
Requirement already satisfied: rich in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmengine>=0.2.0->mmcv==2.0.0rc4) (13.7.1)
Requirement already satisfied: opencv-python>=3 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmengine>=0.2.0->mmcv==2.0.0rc4) (4.8.1.78)
Requirement already satisfied: matplotlib in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmengine>=0.2.0->mmcv==2.0.0rc4) (3.9.1)
Requirement already satisfied: termcolor in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from mmengine>=0.2.0->mmcv==2.0.0rc4) (2.4.0)
Requirement already satisfied: importlib-metadata>=6.6.0 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from yapf->mmcv==2.0.0rc4) (8.2.0)
Requirement already satisfied: platformdirs>=3.5.1 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from yapf->mmcv==2.0.0rc4) (4.2.2)
Requirement already satisfied: tomli>=2.0.1 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from yapf->mmcv==2.0.0rc4) (2.0.1)
Requirement already satisfied: zipp>=0.5 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from importlib-metadata>=6.6.0->yapf->mmcv==2.0.0rc4) (3.19.2)
Requirement already satisfied: contourpy>=1.0.1 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (1.2.1)
Requirement already satisfied: fonttools>=4.22.0 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (4.53.1)
Requirement already satisfied: cycler>=0.10 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (0.12.1)
Requirement already satisfied: kiwisolver>=1.3.1 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (1.4.5)
Requirement already satisfied: python-dateutil>=2.7 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (2.9.0.post0)
Requirement already satisfied: pyparsing>=2.3.1 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (3.1.2)
Requirement already satisfied: markdown-it-py>=2.2.0 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from rich->mmengine>=0.2.0->mmcv==2.0.0rc4) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from rich->mmengine>=0.2.0->mmcv==2.0.0rc4) (2.18.0)
Requirement already satisfied: mdurl~=0.1 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from markdown-it-py>=2.2.0->rich->mmengine>=0.2.0->mmcv==2.0.0rc4) (0.1.2)
Requirement already satisfied: six>=1.5 in e:\comfyui-aki\comfyui-aki-v1.3\python\lib\site-packages (from python-dateutil>=2.7->matplotlib->mmengine>=0.2.0->mmcv==2.0.0rc4) (1.16.0)
Building wheels for collected packages: mmcv
  Building wheel for mmcv (setup.py) ... done
  Created wheel for mmcv: filename=mmcv-2.0.0rc4-cp310-cp310-win_amd64.whl size=4528477 sha256=eb3c981a1a6edde4211523bd1f1790cc02822d36ac8c0c754dfe252cec807c3d
  Stored in directory: C:\Users\Administrator\AppData\Local\Temp\pip-ephem-wheel-cache-5cwh2zk1\wheels\1a\14\d1\793dd16d71f91daf81152095a93d43618e60dee1762c678940
Successfully built mmcv
Installing collected packages: mmcv
  Attempting uninstall: mmcv
    Found existing installation: mmcv 2.2.0
    Uninstalling mmcv-2.2.0:
      Successfully uninstalled mmcv-2.2.0
Successfully installed mmcv-2.0.0rc4

[notice] A new release of pip is available: 23.0.1 -> 24.1.2
[notice] To update, run: python.exe -m pip install --upgrade pip

E:\ComfyUI-aki\ComfyUI-aki-v1.3\python>

2.4、下载并正确组织"权重"路径

2.4.1、下载weight权重文件

下载的位置

2.4.2、"权重"路径的组织

• MuseTalk根路径：

E:\ComfyUI-aki\ComfyUI-aki-v1.3\models\diffusers\TMElyralab\MuseTalk

• MuseTalk的models路径同时部署：

• 其中权重文件79999_iter.pth的版本key值（即文件hash）很重要：

2.5、一键包启动时自动使用权重dw-ll_ucoco_384.pth后端下载checkpoint大模型

    如果“手动安装”成功，开始后端下载checkpoint大模型：

Loads checkpoint by local backend from path:E:\ComfyUI-aki\ComfyUI-aki-v1.3\models\diffusers\TMElyralab/MuseTalk/dwpose/dw-ll_ucoco_384.pth

Downloading: "https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth"
to
E:\ComfyUI-aki\ComfyUI-aki-v1.3.cache\torch\hub\checkpoints\s3fd-619a316812.pth

    如果第一次不成功，就再“一键启动”一次即可。 

2.6、一键包执行

三、配置并执行工作流

wf_sun_musetalk.json （下载后去掉“.txt”扩展名）——

https://www.cpuofbs.com/tests/ai/SD/comfyUI/wf_sun_musetalk.json.txt

四、结果

[https://www.cpuofbs.com/tests/ai/SD/comfyUI/AnimateDiff_00018-audio.mp4]

https://www.cpuofbs.com/tests/ai/SD/comfyUI/AnimateDiff_00018-audio.mp4

ComfyUI-MuseTalk的测试视频

五、本期关联技术博文

《comfyUI-MuseTalk的参数设置》

《AI用Alice_split_toolset切割音频的采样率》

《AI用Alice_split_toolset切割音频的采样率》