FoundationPose复现

目录

intro

论文我就不多说了
code: https://github.com/NVlabs/FoundationPose
paper: https://arxiv.org/abs/2312.08344

1 电脑配置

1. Linux 环境 ：ubuntu 22.04
2. cuda 版本： 11.8
3. 显卡：GeForce RTX 4080
4. IDE：pycharm

2 环境配置

2.1 资源下载

• 库

git clone https://github.com/NVlabs/FoundationPose.git

• 必须下载 权重 主目录下新建./weights/，放在该路径下 测试数据 主目录下新建./demo_data/，放在该路径下
• 可选下载 数据集 太大了，我没下载 model-free few-shot version

2.2 conda环境

这里参考blog

• Eigen3 3.4.0安装

cd$HOME&&wget-q https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz &&\tar-xzf eigen-3.4.0.tar.gz &&\cd eigen-3.4.0 &&mkdir build &&cd build
cmake .. -Wno-dev -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-std=c++14 ..sudomakeinstallcd$HOME&&rm-rf eigen-3.4.0 eigen-3.4.0.tar.gz

建议不要使用官方github上的，我用下面的路径一直配置不对

# Install Eigen3 3.4.0 under conda environment
conda install conda-forge::eigen=3.4.0
exportCMAKE_PREFIX_PATH="$CMAKE_PREFIX_PATH:/eigen/path/under/conda"

• 创建虚拟环境 我这里都是在pycharm的终端中进行（个人习惯）

# create conda environment
conda create -n foundationpose python=3.9# activate conda environment
conda activate foundationpose

• python库

# install dependencies
python -m pip install-r requirements.txt

下面的貌似自动给安装好了

# Install NVDiffRast
python -m pip install--quiet --no-cache-dir git+https://github.com/NVlabs/nvdiffrast.git
# Kaolin (Optional, needed if running model-free setup)
python -m pip install--quiet --no-cache-dir kaolin==0.15.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.0.0_cu118.html
# PyTorch3D
python -m pip install--quiet --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py39_cu118_pyt200/download.html

这里可以检查下自己的torch、pytorch3d版本，终端输入

python

Python 3.9.19 (main, May  62024, 19:43:03)[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license"formore information.
>>>import torch,torchvision,pytorch3d
>>> torch.__version__
'2.0.0+cu118'>>> torchvision.__version__
'0.15.1+cu118'>>> pytorch3d.__version__
'0.7.3'>>> exit()

-构建C++扩展

# Build extensionsCMAKE_PREFIX_PATH=$CONDA_PREFIX/lib/python3.9/site-packages/pybind11/share/cmake/pybind11 bash build_all_conda.sh

这里会花费比较长时间，并且可能会有各种Warning，只要最后能到100%，无视
最后成功的输出

[100%] Linking CXX shared module mycpp.cpython-39-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs[100%] Built target mycpp
Obtaining file:///home/user/code/embodied_AI/FoundationPose/bundlesdf/mycuda
  Preparing metadata (setup.py)... done
Installing collected packages: common
  Running setup.py develop for common
Successfully installed common-0.0.0

3 测试

3.1 初始demo

python run_demo.py

成功弹出

4 补充

4.1 cuda版本问题

我之前的所有代码都是在cuda11.6基础上的，所以这里需要进行一个cuda版本切换，否则会很麻烦（连环报错）
按照blog一步步来

官方下载11.8

wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudosh cuda_11.8.0_520.61.05_linux.run

记得再下载cudnn
更换

~/.bashrc

里的cuda路径并且

source

在虚拟环境终端下

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c)2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_

这里我还在conda中进行了下面操作

 conda installcudatoolkit=11.8

成功

4.2 Build extensions报错

• 报错

bash: build_all_conda.sh: 没有那个文件或目录

原因：路径没有在FoundationPose下

• 遇到找不到Eigen3路径的 在终端使用下面代码重新安装

cd$HOME&&wget-q https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz &&\tar-xzf eigen-3.4.0.tar.gz &&\cd eigen-3.4.0 &&mkdir build &&cd build
cmake .. -Wno-dev -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-std=c++14 ..sudomakeinstallcd$HOME&&rm-rf eigen-3.4.0 eigen-3.4.0.tar.gz

• torch版本对应 如果cuda与torch会pytorch3d的版本不对应，也会报错，所以强烈建议按照官方的版本来，为自己11.6就是遇到各种问题换成11.8直接都解决

4.3 run_demo.py报错

packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/zqy/anaconda3/envs/foundationpose/lib/python3.9/site-packages/torch/nn/modules/transformer.py", line 437, in forward
    return torch._transformer_encoder_layer_fwd(
RuntimeError: expected scalar type Half but found Float

原因：torch的版本太低，切换成2.0以上就不会遇到了