Add BetterTransformer support for FlavaModel by katiele47 · Pull Request #538 · huggingface/optimum

katiele47 · 2022-12-02T18:25:55Z

What does this PR do?

Fixes #20372

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?
[Other] Applied make style

Questions

Should I make this PR (and future PRs that aren't ready for review but are worth discussing) into a draft PR? There are some questions I'd like to confirm with you in advance.
Would it be necessary to add the BetterTransformersFlavaTest or should I reuse BetterTransformersViltTest for FlavaModel? If so, should I update the name to BetterTransformersVisionTextTest?
I ran into the following issues when testing the conversion of FlavaModel with both pytest and running this script

from transformers import AutoModel
from optimum.bettertransformer import BetterTransformer

model_id = "hf-internal-testing/tiny-random-FlavaModel"
model = AutoModel.from_pretrained(model_id)
bt_model = BetterTransformer.transform(model)

Error:

 ~/optimum/optimum/bettertransformer/models/base.py:81 in validate_bettertransformer    │
│                                                                                                  │
│    78 │   │                                                                                      │
│    79 │   │   # Check activation function                                                        │
│    80 │   │   if self.act_fn not in SUPPORTED_ACTIVATION_FUNCTIONS:                              │
│ ❱  81 │   │   │   raise ValueError(                                                              │
│    82 │   │   │   │   f"Activation function {self.act_fn} not supported" " for `BetterTransfor   │
│    83 │   │   │   )                                                                              │
│    84 │   │   self.use_gelu = (self.act_fn == "gelu") or (self.act_fn == "gelu_new")             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Activation function None not supported for `BetterTransformer` integration.

I couldn't find where the self.act_fn get defined inside the forward method after following this guide. Would appreciate any help!

younesbelkada

Hi @katiele47 !
Thanks so much for your very clean and great PR! Everything looks perfect 🔥
Regarding FLAVA, I quickly dived into it and the fix seems to be relatively simple. The model has 2 components, image and text encoders. Each of these submodules has its own config file that you can access with .image_config or .text_config. I think that the fix that I propose here is fine, as for both sub-configs the same hyper parameters seems to be used (i.e. same hidden_size, same hidden_act, etc.). Check for example this config file

After accepting my suggestion, you can run pytest tests/bettertransformer/test_bettertransformer_vision.py:: BetterTransformersFlavaTest to make sure the test pass ;)

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

katiele47 · 2022-12-03T05:37:34Z

Thanks a lot for your insight @younesbelkada! I will run the test and update this PR

…d-better-transformers-support-for-flava

…ithub.com/katiele47/optimum into add-better-transformers-support-for-flava

katiele47 · 2022-12-03T19:56:41Z

The test script now worked with the following output, but I encountered 2 failed tests from BetterTransformersViltTest and BetterTransformerFlavaTest. Do you think I should use a different resized image here? It seems weird how the BetterTransformersViltTest previously passed with the current input image.

FAILED tests/bettertransformer/test_bettertransformer_vision.py::BetterTransformersViLTTest::test_logits - ValueError: Input image size (224*224) doesn't match model (30*30).
FAILED tests/bettertransformer/test_bettertransformer_vision.py::BetterTransformersFlavaTest::test_logits - ValueError: Input image size (224*224) doesn't match model (30*30).

Test script output:

FlavaModel(
  (text_model): FlavaTextModel(
    (embeddings): FlavaTextEmbeddings(
      (word_embeddings): Embedding(1124, 32, padding_idx=0)
      (position_embeddings): Embedding(512, 32)
      (token_type_embeddings): Embedding(2, 32)
      (LayerNorm): LayerNorm((32,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.0, inplace=False)
    )
    (encoder): FlavaEncoder(
      (layer): ModuleList(
        (0): FlavaLayerBetterTransformer()
        (1): FlavaLayerBetterTransformer()
        (2): FlavaLayerBetterTransformer()
        (3): FlavaLayerBetterTransformer()
        (4): FlavaLayerBetterTransformer()
      )
    )
    (layernorm): LayerNorm((32,), eps=1e-12, elementwise_affine=True)
    (pooler): FlavaPooler(
      (dense): Linear(in_features=32, out_features=32, bias=True)
      (activation): Tanh()
    )
  )
  (image_model): FlavaImageModel(
    (embeddings): FlavaImageEmbeddings(
      (patch_embeddings): PatchEmbeddings(
        (projection): Conv2d(3, 32, kernel_size=(2, 2), stride=(2, 2))
      )
      (dropout): Dropout(p=0.0, inplace=False)
    )
    (encoder): FlavaEncoder(
      (layer): ModuleList(
        (0): FlavaLayerBetterTransformer()
        (1): FlavaLayerBetterTransformer()
        (2): FlavaLayerBetterTransformer()
        (3): FlavaLayerBetterTransformer()
        (4): FlavaLayerBetterTransformer()
      )
    )
    (layernorm): LayerNorm((32,), eps=1e-12, elementwise_affine=True)
    (pooler): FlavaPooler(
      (dense): Linear(in_features=32, out_features=32, bias=True)
      (activation): Tanh()
    )
  )
  (multimodal_model): FlavaMultimodalModel(
    (encoder): FlavaEncoder(
      (layer): ModuleList(
        (0): FlavaLayerBetterTransformer()
        (1): FlavaLayerBetterTransformer()
        (2): FlavaLayerBetterTransformer()
        (3): FlavaLayerBetterTransformer()
        (4): FlavaLayerBetterTransformer()
      )
    )
    (layernorm): LayerNorm((32,), eps=1e-12, elementwise_affine=True)
    (pooler): FlavaPooler(
      (dense): Linear(in_features=32, out_features=32, bias=True)
      (activation): Tanh()
    )
  )
  (image_projection): Linear(in_features=32, out_features=32, bias=True)
  (text_projection): Linear(in_features=32, out_features=32, bias=True)
  (image_to_mm_projection): Linear(in_features=32, out_features=32, bias=True)
  (text_to_mm_projection): Linear(in_features=32, out_features=32, bias=True)
)

younesbelkada

Thanks so much for digging into the issue!
I think that there is a problem with the feature extractor in hf-internal-testing/tiny-random-FlavaModel , can you please try with a new model that I created? (Suggestion below) Thanks!

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

…d-better-transformers-support-for-flava

…ithub.com/katiele47/optimum into add-better-transformers-support-for-flava

HuggingFaceDocBuilderDev · 2022-12-05T22:40:48Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

katiele47 · 2022-12-05T22:50:00Z

Thanks again for your time and suggestion @younesbelkada! I have made the changes accordingly, and all tests passed. I'll keep an eye on any arising CircleCI issues in the meantime.

younesbelkada

Hi @katiele47 !
I left a couple of comments, this should fix the failing tests! ;)
Thanks again

* ORTQuantizer saves model config when possible * Fix docstring * ORTModel can load any name for the ONNX model * Use regex now * Almost ready * modeling_ort * test * Works for ORTModel * Fixed bugs * remove pdb * Remove unnecessary attribute * ORTDecoderModel ready * Fix issue * Make style * fix issues * Rebase and style * Fix * Fix issues * Apply suggestions Co-authored-by: Michael Benayoun <michael@huggingface.co>

* Add IO binding for ORTModelForCustomTasks * Add test * Fix test model * Force inputs to be contiguous * Fix docstring * Nits

* Update README * Change section * Add links to examples * Modify BetterTransformer documentation section name * Add link to the documentation * Rephrasing * Update README.md Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com> * Update readme Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>

* Fix issues * Fix issues * Apply suggestion Co-authored-by: Michael Benayoun <michael@huggingface.co>

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

…ransformers 4.25.1 (huggingface#561) * Fix wrap * Improve import check * Improve import check

* add clip * style and test * better support * debug * style * support clip vision model * typo * fix test * fix test * add clip in doc * fix test * remove comment * Update optimum/bettertransformer/models/encoder_models.py Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * fix style Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* fix flaky test * fix flaky

…nx` in `ORTDecoder` (huggingface#554) * add support * cleaning * remove useless import * remove print * styling post merge * Update optimum/onnxruntime/modeling_decoder.py Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com> * simplify multiple ONNX export * fix path * avoid code duplicate, better doc * fix and simplify test Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com>

* enable FP16Optimizer for fp16 deepspeed training * wrap apex optimizer * Update optimum/onnxruntime/trainer.py Co-authored-by: Jingya HUANG <44135271+JingyaHuang@users.noreply.github.com> * Update optimum/onnxruntime/trainer.py Co-authored-by: Jingya HUANG <44135271+JingyaHuang@users.noreply.github.com> * formating trainer.py * reformat using black 22.6 Co-authored-by: Jingya HUANG <44135271+JingyaHuang@users.noreply.github.com> Co-authored-by: Adam Louly <adamlouly@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>

katiele47 · 2022-12-09T20:02:38Z

@younesbelkada Sorry for the multiple file changes, I think it's due to merging conflicts with upstream main, but I'm not entirely sure. Please let me know if this affect the original flava changes.

katiele47 · 2023-03-22T03:04:20Z

Hi @younesbelkada I have created a new PR here that replaces this outdated PR. Please review and let me know if I should close this one. Thanks a lot!

add BetterTransformer support for FlavaModel and test

e2fa9b4

younesbelkada reviewed Dec 2, 2022

View reviewed changes

Comment thread optimum/bettertransformer/models/encoder_models.py Outdated

younesbelkada mentioned this pull request Dec 2, 2022

Community contribution - BetterTransformer integration for more models! huggingface/transformers#20372

Closed

14 tasks

Update optimum/bettertransformer/models/encoder_models.py

c76b1c4

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

Bibi added 2 commits December 3, 2022 14:33

Merge branch 'main' of https://github.com/huggingface/optimum into ad…

fadd2ed

…d-better-transformers-support-for-flava

Merge branch 'add-better-transformers-support-for-flava' of https://g…

f91a928

…ithub.com/katiele47/optimum into add-better-transformers-support-for-flava

younesbelkada reviewed Dec 5, 2022

View reviewed changes

Comment thread tests/bettertransformer/test_bettertransformer_vision.py Outdated

katiele47 and others added 3 commits December 5, 2022 16:54

Update tests/bettertransformer/test_bettertransformer_vision.py

072420c

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

Merge branch 'main' of https://github.com/huggingface/optimum into ad…

4966b96

…d-better-transformers-support-for-flava

Merge branch 'add-better-transformers-support-for-flava' of https://g…

6c80658

…ithub.com/katiele47/optimum into add-better-transformers-support-for-flava

younesbelkada reviewed Dec 6, 2022

View reviewed changes

Comment thread tests/bettertransformer/test_bettertransformer_vision.py Outdated

Comment thread tests/bettertransformer/test_bettertransformer_vision.py

michaelbenayoun and others added 15 commits December 6, 2022 10:13

Add IO binding support for custom ORTModel (huggingface#447)

8b559db

* Add IO binding for ORTModelForCustomTasks * Add test * Fix test model * Force inputs to be contiguous * Fix docstring * Nits

fix import (huggingface#553)

d026fdc

Refactor of 2 functions used in ORTModel (huggingface#551)

f6eb417

* Fix issues * Fix issues * Apply suggestion Co-authored-by: Michael Benayoun <michael@huggingface.co>

Update tests/bettertransformer/test_bettertransformer_vision.py

382077d

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

Update tests/bettertransformer/test_bettertransformer_vision.py

422f3d7

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

Update readme (huggingface#556)

fac2694

applied make style

08d7917

Fix ORTTrainer wrapper duplication / PyTorch evaluate / update with t…

6063fc4

…ransformers 4.25.1 (huggingface#561) * Fix wrap * Improve import check * Improve import check

fix test

d169fc3

Fix flaky BetterTransformer test (huggingface#564)

86375c4

* fix flaky test * fix flaky

Bibi added 4 commits December 9, 2022 14:32

fixed merge conflict due to rebase with upstream main

1521f1b

merge conflict clip and flava

f48da36

installed missing dependencies

0b8ed50

applied make style

a012ec7

katiele47 mentioned this pull request Mar 22, 2023

Flava model better transformers #907

Closed

3 tasks

fxmarty closed this Aug 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BetterTransformer support for FlavaModel #538

Add BetterTransformer support for FlavaModel #538
katiele47 wants to merge 26 commits into
huggingface:mainfrom
katiele47:add-better-transformers-support-for-flava

katiele47 commented Dec 2, 2022 •

edited

Loading

Uh oh!

younesbelkada left a comment

Uh oh!

Uh oh!

katiele47 commented Dec 3, 2022

Uh oh!

katiele47 commented Dec 3, 2022

Uh oh!

younesbelkada left a comment

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Dec 5, 2022

Uh oh!

katiele47 commented Dec 5, 2022

Uh oh!

younesbelkada left a comment

Uh oh!

Uh oh!

Uh oh!

katiele47 commented Dec 9, 2022

Uh oh!

katiele47 commented Mar 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

katiele47 commented Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Questions

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

katiele47 commented Dec 3, 2022

Uh oh!

katiele47 commented Dec 3, 2022

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Dec 5, 2022

Uh oh!

katiele47 commented Dec 5, 2022

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

katiele47 commented Dec 9, 2022

Uh oh!

katiele47 commented Mar 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

katiele47 commented Dec 2, 2022 •

edited

Loading