It seems like the rng_state is not changing after a call of a random method.

[dhorka]

Hi,

I was just trying to figure out how pytorch lightning manage the rng_states and I found something that I do not completly understand. What I am trying to check if that after a call of torch.rand() the rng_state of torch_lightning is increased. In plain torch the test is something like this:

>>> import torch
>>> torch.manual_seed(123)
<torch._C.Generator object at 0x7f2144d6e948>
>>> init_state = torch.get_rng_state()
>>> n = torch.rand(1)
>>> state = torch.get_rng_state()
>>> init_state.sum()
tensor(317672)
>>> state.sum()
tensor(322088)

As expected the state has changed after a call of a random method. Then I tried to reproduce the same experiment on torch lightning using this code:

import pytorch_lightning as pl    
import torch    
      
      
  class TestModule(pl.LightningModule):    
      def __init__(self):    
          super(TestModule, self).__init__()    
          self.nn = torch.nn.Linear(3, 500)    
      
      def training_step(self, batch, batch_indx):    
          init_state = torch.get_rng_state()    
          n = torch.rand(1)    
          state = torch.get_rng_state()    
          print('Init state: {}, last state: {}'.format(init_state.sum(), state.sum()))    
                                                                                                                                                                                                  
      def configure_optimizers(self):                                                                                                                                                             
          print('Configuring optimizers')                                                                                                                                                         
          return torch.optim.Adam(self.parameters(), lr=0.001)                                                                                                                                    
                                                                                                                                                                                                  
      def setup(self, stage):
          x = torch.ones(2, 1)  
          self.train_data = torch.utils.data.TensorDataset(x)
          # print('setup on rank: {} data: {}'.format(self.global_rank, x))
                                                                               
      def train_dataloader(self):
          return torch.utils.data.DataLoader(self.train_data, batch_size=2, num_workers=2, shuffle=0)
                                                                                              
      
  if __name__ == '__main__':
                                
      pl.seed_everything(123)
                                 
      net = TestModule()
      trainer = pl.Trainer(gpus=0, max_epochs=2)
                                                    
      trainer.fit(net)

And the output is this one:

Global seed set to 123
GPU available: True, used: False
TPU available: None, using: 0 TPU cores
Configuring optimizers

  | Name | Type   | Params
--------------------------------
0 | nn   | Linear | 2.0 K
--------------------------------
2.0 K     Trainable params
0         Non-trainable params
2.0 K     Total params
0.008     Total estimated model params size (MB)
Epoch 0:  Init state: 316114, last state: 316114
Epoch 1:  Init state: 316114, last state: 316114

As you can see in that case the random generator state has not changed and I do not understand the reason for that. Am I missing something?

[dhorka]

I found that the problem was doing the sum. If i do the manual_seed before doing the random call I obtain the same results as in vanilla pytorch.