Add RoCBert support for Bettertransformer by shogohida · Pull Request #542 · huggingface/optimum

shogohida · 2022-12-03T14:15:57Z

What does this PR do?

Adds RoCBert support for Bettertransformer

Fixes huggingface/transformers#20372

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2022-12-03T15:01:48Z

The documentation is not available anymore as the PR was closed or merged.

shogohida · 2022-12-03T16:24:42Z

I got the following error but I didn't get it....

https://github.com/huggingface/optimum/actions/runs/3608885843/jobs/6081897113
https://github.com/huggingface/optimum/actions/runs/3608885843/jobs/6081897157

 File "/Users/runner/work/optimum/optimum/tests/bettertransformer/testing_bettertransformer_utils.py", line 110, in test_raise_autocast
    _ = bt_model(**inputs)
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/transformers/models/roc_bert/modeling_roc_bert.py", line 1041, in forward
    embedding_output = self.embeddings(
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/transformers/models/roc_bert/modeling_roc_bert.py", line 282, in forward
    embedding_in = self.LayerNorm(embedding_in)
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type BFloat16 but found Float

younesbelkada

Thanks so much for adding BetterTransformer support for this new architecture!
SInce ROCBert copies exactly the same structure as Bert, we can leverage that and use directly BertLayerBetterTransformer as the converted layer :D
Regarding the failing test, let me dig a little bit and get back to you

shogohida · 2022-12-05T13:02:30Z

Hi @younesbelkada
Thanks for your comment! I deleted RoCBertLayerBetterTransformer as you indicated. I received another error which is following.

FAILED onnxruntime/test_optimization.py::ORTOptimizerTest::test_compare_original_seq2seq_model_with_optimized_model_7 - FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/.cache/huggingface/hub/hf-internal-testing/tiny-random-onnx-mt5/decoder_model_o2_cpu.onnx'

younesbelkada · 2022-12-05T19:24:14Z

Hi @shogohida
Don't worry about this test as it is flaky ;) Will look into the failing BT test now

shogohida · 2022-12-07T16:10:52Z

There is another error that I don't understand...

ERROR: test_inference_speed (test_bettertransformer_encoder.BetterTransformersEncoderTest)
The converted models should be at least slightly faster than the native
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/runner/work/optimum/optimum/tests/bettertransformer/test_bettertransformer_encoder.py", line 169, in test_inference_speed
    _ = bt_model(input_ids, attention_mask=attention_mask)
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 1034, in forward
    pooled_output = self.pooler(sequence_output) if self.pooler is not None else None
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 661, in forward
    first_token_tensor = hidden_states[:, 0]
IndexError: index 0 is out of bounds for dimension 1 with size 0

fxmarty · 2022-12-08T11:57:14Z

@shogohida Thanks for the work! This is a flaky test, fixed in #564 . I rerun the workflow and it should be fine.

fxmarty · 2022-12-08T14:29:48Z

Yet an other test failing:

======================================================================
ERROR: test_raise_autocast (test_bettertransformer_encoder.BetterTransformersEncoderTest)
A tests that checks if the conversion raises an error if the model is run under
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/optimum/optimum/tests/bettertransformer/testing_bettertransformer_utils.py", line 141, in test_raise_autocast
    _ = bt_model(**inputs)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/transformers/models/roc_bert/modeling_roc_bert.py", line 1041, in forward
    embedding_output = self.embeddings(
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/transformers/models/roc_bert/modeling_roc_bert.py", line 282, in forward
    embedding_in = self.LayerNorm(embedding_in)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type BFloat16 but found Float

younesbelkada · 2022-12-08T20:01:43Z

This failing test has to do with autocast and ROCBert. I think that the fix should go on transformers side or we can just skip this test for RocBert..

shogohida · 2022-12-09T09:21:44Z

Thanks for your comments guys! So what should I do?.... Can we skip the failing test as Younes said?

younesbelkada · 2022-12-09T09:24:42Z

Hi @shogohida
Thanks so much for the heads up!
Yes let's skip the test for now. can you create a separate testing class for ROCBert (check what we have done for Vilt for example) Let me know if I can help !

shogohida · 2022-12-09T09:58:25Z

@younesbelkada
Sure! I need to create a separate test class like this, right?

I'll let you know if I get stuck!

younesbelkada · 2022-12-09T09:59:22Z

This is correct, please proceed as suggested ;-)

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

shogohida · 2023-01-08T02:02:21Z

@younesbelkada
Sorry for coding this late. I added a separate test class for RocBert but I didn't know how to change prepare_inputs_for_class... I read other parts of the code but I couldn't find the solution and just copied from BetterTransformersEncoderTest.

61ff473

fxmarty · 2023-01-08T18:50:48Z

Hi @shogohida , thanks for working on it! Maybe

class BetterTransformersRoCBertTest(BetterTransformersEncoderTest):
    all_models_to_test = ["path-to-tiny-rocbert-model-here"]
    
    # unrelated issue with torch.amp.autocast with rocbert (expected scalar type BFloat16 but found Float)
    def test_raise_autocast(self):
        pass

would work

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

shogohida · 2023-01-08T22:03:16Z

Hi @fxmarty
Thanks for your comment! I changed the code as you indicated.

228aa15

It seems I removed a review request for you but it was an accident when I requested a review from Younes 🙏

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

shogohida · 2023-01-10T22:30:06Z

Still facing the same error....

https://github.com/huggingface/optimum/actions/runs/3882552677/jobs/6630181175
https://github.com/huggingface/optimum/actions/runs/3882552677/jobs/6630181346

Traceback (most recent call last):
  File "/Users/runner/work/optimum/optimum/tests/bettertransformer/testing_bettertransformer_utils.py", line 141, in test_raise_autocast
    _ = bt_model(**inputs)
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/transformers/models/roc_bert/modeling_roc_bert.py", line 1041, in forward
    embedding_output = self.embeddings(
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/transformers/models/roc_bert/modeling_roc_bert.py", line 282, in forward
    embedding_in = self.LayerNorm(embedding_in)
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/Users/runner/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/torch/nn/functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type BFloat16 but found Float

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

fxmarty · 2023-01-12T15:49:19Z

Thank you for your contribution!

shogohida · 2023-01-12T23:20:31Z

Thanks for your review! It took time but it was merged in the end... This was my first issue so I hope to contribute more!

shogohida marked this pull request as ready for review December 3, 2022 14:16

younesbelkada requested a review from fxmarty December 3, 2022 14:44

younesbelkada reviewed Dec 5, 2022

View reviewed changes

Comment thread optimum/bettertransformer/models/encoder_models.py Outdated

Comment thread optimum/bettertransformer/models/__init__.py Outdated

Comment thread optimum/bettertransformer/models/__init__.py Outdated

younesbelkada mentioned this pull request Dec 5, 2022

Community contribution - BetterTransformer integration for more models! huggingface/transformers#20372

Closed

14 tasks

fxmarty mentioned this pull request Dec 8, 2022

Fix flaky BetterTransformer test #564

Merged

fxmarty closed this in #564 Dec 8, 2022

fxmarty reopened this Dec 8, 2022

shogohida added 6 commits January 8, 2023 09:19

Add RoCBert to overview doc

f7aad13

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

Add RoCBertLayerBetterTransformer

4cd8034

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

Add RoCBertLayerBetterTransformer to init

558ea42

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

Add tiny-random-RoCBertModel

5d3fa65

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

Run make style

42140ff

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

Delete RoCBertLayerBetterTransformer

92f024f

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

shogohida force-pushed the add-rocbert-support-for-bettertransformer branch from 74af5d4 to 92f024f Compare January 8, 2023 00:55

shogohida added 2 commits January 8, 2023 10:01

Add rocbert to init

6c09f51

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

Fix test

61ff473

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

shogohida mentioned this pull request Jan 8, 2023

Tranformers documentation translation to Japanese 🇯🇵 huggingface/transformers#18413

Closed

10 tasks

Fix test

228aa15

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

shogohida requested review from younesbelkada and removed request for fxmarty January 8, 2023 22:04

Change rocbert to roc_bert

97d9538

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

fxmarty reviewed Jan 11, 2023

View reviewed changes

Comment thread tests/bettertransformer/test_bettertransformer_encoder.py Outdated

Remove tiny-random-RoCBertModel

4c7925e

Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

fxmarty merged commit b412390 into huggingface:main Jan 12, 2023

shogohida deleted the add-rocbert-support-for-bettertransformer branch January 12, 2023 23:19

viktor-shcherb mentioned this pull request Apr 5, 2023

Enable AMP for BetterTransformer at torch >= 2.0.0 #953

Closed

3 tasks

Conversation

shogohida commented Dec 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Uh oh!

HuggingFaceDocBuilderDev commented Dec 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shogohida commented Dec 3, 2022

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shogohida commented Dec 5, 2022

Uh oh!

younesbelkada commented Dec 5, 2022

Uh oh!

shogohida commented Dec 7, 2022

Uh oh!

fxmarty commented Dec 8, 2022

Uh oh!

fxmarty commented Dec 8, 2022

Uh oh!

younesbelkada commented Dec 8, 2022

Uh oh!

shogohida commented Dec 9, 2022

Uh oh!

younesbelkada commented Dec 9, 2022

Uh oh!

shogohida commented Dec 9, 2022

Uh oh!

younesbelkada commented Dec 9, 2022

Uh oh!

shogohida commented Jan 8, 2023

Uh oh!

fxmarty commented Jan 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shogohida commented Jan 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shogohida commented Jan 10, 2023

Uh oh!

Uh oh!

fxmarty commented Jan 12, 2023

Uh oh!

shogohida commented Jan 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shogohida commented Dec 3, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 3, 2022 •

edited

Loading

fxmarty commented Jan 8, 2023 •

edited

Loading

shogohida commented Jan 8, 2023 •

edited

Loading