Skip to content
This repository was archived by the owner on Aug 7, 2024. It is now read-only.

Set amax_and_scale_synced unconditionally #220

Closed
wants to merge 3 commits into from
Closed

Set amax_and_scale_synced unconditionally #220

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
77d785e
Set `amax_and_scale_synced` unconditionally
Feb 16, 2024
075b8cb
Update on "Set `amax_and_scale_synced` unconditionally"
Feb 16, 2024
f438dea
Update on "Set `amax_and_scale_synced` unconditionally"
Feb 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions float8_experimental/float8_linear_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,5 @@ def sync_float8_amax_and_scale_history(model: torch.nn.Module, fp8_layers=None)
child.fp8_scale_w.copy_(new_w_scales[idx])
child.fp8_scale_dL_dY.copy_(new_dL_dY_scales[idx])

# 4. set a flag to signal amaxes/scales are ready
# We only update the flag if we know it will be checked by the modules
if fp8_config.enable_amax_init:
child.amax_and_scale_synced = True
# Set a flag to signal amaxes/scales are ready
child.amax_and_scale_synced = True