AutoML 2.0: Should Seed be a tunable param?

[torronen]

One of the tunable parameters is FeatureFraction. Per my understanding this drops randomly the specified portion of columns for each tree. Further, I understand that random seed is used to decide which columns to drop.

In small forests with few trees the specific feature columns that are dropped may have bigger impact than how many features are dropped. One of the columns may lead to overfit and dropping it may improve the results on validation and test sets. However, it is crucial that the correct columns is dropped.

If my understanding is correct, then I believe seed should be one of the tunable parameters. This should be the case at least for algorithms where FeatureFraction is present (FastTree and LightGBM at least). At the moment, it is not present in e.g. FastTreeOption class.

Agree/disagree?

[LittleLittleCloud]

That's an interesting point. Theoretically I agree with you that we should put seed as a tunable parameter as seed affects the final result. But I also have the concern on whether introducing seed into the search space would bring more benefits than trouble. As the way of how seed influences training metric is not continuous and highly randomness, it might be not worth to bring it into search space.

Let me run a few experiments which includes seed into search space and see how it works.

[torronen]

Right, makes sense. If it were only a choice of, let's say, 3 seeds (1,2,3), in that case it could prevent getting the worst random sequence in case the user is unlucky. Finding optimal seed could be a slow search without possivility to optimize the search. There would probably be faster ways to decide which columns to drop, and in which order.