Simplify keep_in_fp32_modules logic #36722
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
Nice! One thing that is missing: a check to enforce at init time, like the TP plan, that the layers actually exist in the model?
I think for Blip2 and T5 there might be weirdness, but at least show a warning !
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
Also some documentation is missing + maybe deprecate nion re pattern no?
|
I just modified blip2 so that no model in the library can have the flag set to a non-existing layer (😮💨), so I don't think we actually need a check at init time - IMO it would clutter the code for something that would never happen. It can add it to be super super safe though! |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
LGTM, IMO from_pretrained should be checking the layer names to make sure typos are avoided for example
Cyrilvallez
force-pushed
the
fix-keep-fp32
branch
from
ba157c2 to
4db38b0
Compare
* better regex everywhere * fix * Update test_modeling_instructblip.py * BC with explanations this time otherwise it makes no sense at all * Update test_modeling_instructblip.py * style * CIs * update _keep_in_fp32_modules in blip2 * Update modeling_utils.py * Update modeling_utils.py * style * CIs * add check * trigger CIs * Update modeling_utils.py * trigger CIs
What does this PR do?
As per the title. It's easier to use a regex from the beginning compared to start with a list, then switch to regex near the end of the loading logic.
Also, make sure that only full layer names can be matched with the regex. Otherwise some old and bad layer names such as
woin T5 could be match against layers such asword_embedding, which containswo.Also, use the flag only when loading in fp16 or using a quantizer expecting it, as
_keep_in_fp32_moduleswas introduced only to avoid issues when casting bf16 -> fp16. See #20287 for details. That is, the layers should not always be kept in fp32, despite the name of the flag... I added a detailed comment about that because it is not clear at all otherwise