Skip to content
This repository was archived by the owner on Aug 7, 2024. It is now read-only.

Set amax_and_scale_synced unconditionally #220

Closed
wants to merge 3 commits into from
Closed

Conversation

awgu
Copy link

@awgu awgu commented Feb 16, 2024

Stack from ghstack (oldest at bottom):

Float8Linear.amax_and_scale_synced is orthogonal to config.enable_amax_init.

  • enable_amax_init = False skips the lazy initialization of amaxes and scales. Without this, the 1st iteration's amaxes will be the default (E4M3_MAX_POS).
  • amax_and_scale_synced is checked every pre-forward regardless of config.enable_amax_init.

if (
self.is_amax_initialized
and (not self.amax_and_scale_synced)
and torch.is_grad_enabled()
):
raise AssertionError(
"amaxes and scales not synced, please call `sync_float8_amax_and_scale_history` before forward"
)

Andrew Gu added 2 commits February 16, 2024 07:36
`Float8Linear.amax_and_scale_synced` is orthogonal to `config.enable_amax_init`.
- `enable_amax_init = False` skips the lazy initialization of amaxes and scales. Without this, the 1st iteration's amaxes will be the default (`E4M3_MAX_POS`).
- `amax_and_scale_synced` is checked every pre-forward regardless of `config.enable_amax_init`.

https://github.com/pytorch-labs/float8_experimental/blob/956195bbc3983594308e7ccd64bab204c457a7a8/float8_experimental/float8_linear.py#L266-L273

[ghstack-poisoned]
@awgu awgu requested a review from drisspg February 16, 2024 16:02
@drisspg
Copy link
Contributor

drisspg commented Feb 16, 2024

So I added this because this module mutation was causing a graph break, hmmm

@awgu
Copy link
Author

awgu commented Feb 16, 2024

@drisspg Would it be possible to define an inner function and compile that? That might also help with any other graph break issues, e.g. like the fp8_layers comment.

facebook-github-bot pushed a commit that referenced this pull request Feb 22, 2024
Summary:
See this comment: #220 (comment)

I still need to verify that this works, I also want to add a test for when we compile the outer func.. not sure what it happends
cc awgu, bdhirsh

Pull Request resolved: #221

Reviewed By: bdhirsh

Differential Revision: D54074075

Pulled By: drisspg

fbshipit-source-id: 185dd60d39e866122f55cef78d1fba9b475088a4
@awgu awgu closed this Feb 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants