running https://github.com/huggingface/accelerate/blob/main/examples/torch_native_parallelism/nd_parallel_trainer.py as accelerate launch --config-file configs/tp_hsdp.yaml nd_parallel_trainer.py. crash like
[rank1]: Traceback (most recent call last):
[rank1]: File "/mnt/disk3/wangyi/accelerate/examples/torch_native_parallelism/nd_parallel_trainer.py", line 82, in
[rank1]: main()
[rank1]: File "/mnt/disk3/wangyi/accelerate/examples/torch_native_parallelism/nd_parallel_trainer.py", line 77, in main
[rank1]: trainer.train()
[rank1]: File "/mnt/disk3/wangyi/transformers/src/transformers/trainer.py", line 2174, in train
[rank1]: return inner_training_loop(
[rank1]: ^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/mnt/disk3/wangyi/transformers/src/transformers/trainer.py", line 2339, in _inner_training_loop
[rank1]: self.optimizer = self.accelerator.prepare(self.optimizer)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/mnt/disk3/wangyi/accelerate/src/accelerate/accelerator.py", line 1515, in prepare
[rank1]: raise ValueError(
[rank1]: ValueError: When using FSDP2, a model and optimizer must be passed together to Accelerator.prepare() as the optimizer needs to have its parameters modified after the model is converted.

making FSDP2 + TP integration smoother is in the roadmap. Almost done with TP, going to tackle FSDP2 next

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43226&sha=754498

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.