You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[rank0]: ValueError: Model architectures ['Qwen2ForCausalLM'] failed to be inspected. Please check the logs for more details. #6
[INFO|image_processing_auto.py:300] 2025-02-13 15:12:53,819 >> Could not locate the image processor configuration file, will try to use the model config instead.
ERROR 02-13 15:12:54 registry.py:299] Error in inspecting model architecture 'Qwen2ForCausalLM'
ERROR 02-13 15:12:54 registry.py:299] Traceback (most recent call last):
ERROR 02-13 15:12:54 registry.py:299] File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 495, in _run_in_subprocess
ERROR 02-13 15:12:54 registry.py:299] returned.check_returncode()
ERROR 02-13 15:12:54 registry.py:299] File "/home/liu/anaconda3/lib/python3.11/subprocess.py", line 502, in check_returncode
ERROR 02-13 15:12:54 registry.py:299] raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 02-13 15:12:54 registry.py:299] subprocess.CalledProcessError: Command '['/home/liu/anaconda3/bin/python', '-m', 'vllm.model_executor.models.registry']' returned non-zero exit status 1.
ERROR 02-13 15:12:54 registry.py:299]
ERROR 02-13 15:12:54 registry.py:299] The above exception was the direct cause of the following exception:
ERROR 02-13 15:12:54 registry.py:299]
ERROR 02-13 15:12:54 registry.py:299] Traceback (most recent call last):
ERROR 02-13 15:12:54 registry.py:299] File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 297, in _try_inspect_model_cls
ERROR 02-13 15:12:54 registry.py:299] return model.inspect_model_cls()
ERROR 02-13 15:12:54 registry.py:299] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-13 15:12:54 registry.py:299] File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 268, in inspect_model_cls
ERROR 02-13 15:12:54 registry.py:299] return _run_in_subprocess(
ERROR 02-13 15:12:54 registry.py:299] ^^^^^^^^^^^^^^^^^^^
ERROR 02-13 15:12:54 registry.py:299] File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 498, in _run_in_subprocess
ERROR 02-13 15:12:54 registry.py:299] raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 02-13 15:12:54 registry.py:299] RuntimeError: Error raised in subprocess:
ERROR 02-13 15:12:54 registry.py:299] Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp-a34b3233.so.1 library.
ERROR 02-13 15:12:54 registry.py:299] Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
ERROR 02-13 15:12:54 registry.py:299]
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/liu/work/X-R1/src/x_r1/grpo.py", line 182, in <module>
[rank0]: main(script_args, training_args, model_args)
[rank0]: File "/home/liu/work/X-R1/src/x_r1/grpo.py", line 134, in main
[rank0]: trainer = XGRPOTrainer(
[rank0]: ^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 369, in __init__
[rank0]: self.llm = LLM(
[rank0]: ^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/utils.py", line 1039, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 240, in __init__
[rank0]: self.llm_engine = self.engine_class.from_engine_args(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 479, in from_engine_args
[rank0]: engine_config = engine_args.create_engine_config(usage_context)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 1059, in create_engine_config
[rank0]: model_config = self.create_model_config()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 983, in create_model_config
[rank0]: return ModelConfig(
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/config.py", line 348, in __init__
[rank0]: self.multimodal_config = self._init_multimodal_config(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/config.py", line 408, in _init_multimodal_config
[rank0]: if ModelRegistry.is_multimodal_model(architectures):
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 433, in is_multimodal_model
[rank0]: model_cls, _ = self.inspect_model_cls(architectures)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 393, in inspect_model_cls
[rank0]: return self._raise_for_unsupported(architectures)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/liu/.local/lib/python3.11/site-packages/vllm/model_executor/models/registry.py", line 350, in _raise_for_unsupported
[rank0]: raise ValueError(
[rank0]: ValueError: Model architectures ['Qwen2ForCausalLM'] failed to be inspected. Please check the logs for more details.
[rank0]:[W213 15:12:55.655552884 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
W0213 15:12:56.503000 329696 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 329799 closing signal SIGTERM
W0213 15:12:56.504000 329696 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 329800 closing signal SIGTERM
W0213 15:12:56.504000 329696 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 329801 closing signal SIGTERM
E0213 15:12:56.834000 329696 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 329798) of binary: /home/liu/anaconda3/bin/python
Traceback (most recent call last):
File "/home/liu/.local/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/liu/.local/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/liu/.local/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1157, in launch_command
deepspeed_launcher(args)
File "/home/liu/.local/lib/python3.11/site-packages/accelerate/commands/launch.py", line 845, in deepspeed_launcher
distrib_run.run(args)
File "/home/liu/anaconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/home/liu/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/liu/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
src/x_r1/grpo.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-13_15:12:56
host : ubuntu-SYS-4029GP-TRT
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 329798)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Activity
dhcode-cpp commentedon Feb 13, 2025
支持3090。 请提供更多错误信息
magnificent1208 commentedon Feb 13, 2025
magnificent1208 commentedon Feb 13, 2025
magnificent1208 commentedon Feb 13, 2025
magnificent1208 commentedon Feb 13, 2025
magnificent1208 commentedon Feb 13, 2025
标题所述问题，我通过添加了这行命令，解决了
export MKL_THREADING_LAYER=GNU以下内容是搜索提供的解释：
根本原因：mkl - service 库的 MKL_THREADING_LAYER=INTEL 设置与 libgomp - a34b3233.so.1 库不兼容。这使得在 vllm.model_executor.models.registry.py 中尝试通过子进程检查 Qwen2ForCausalLM 模型架构时，子进程返回非零退出状态，最终导致 ValueError，阻止了 XGRPOTrainer 的正确初始化。
解决方法：
调整 MKL_THREADING_LAYER 设置：尝试将 MKL_THREADING_LAYER 设置为与 libgomp 兼容的值，如 GNU。在运行脚本前，通过终端设置环境变量：
dhcode-cpp commentedon Feb 13, 2025
已解决，按照readme安装环境。