Estimating time required for PFI

[torronen]

Problem
Running PFI takes time. Users can set permutation count and number of examples to use to adjust accuracy vs. speed. The model, number of features and examples affect the time PFI takes to complete from less than a hour to multiple days. A typical case would be that developer has an allowed timeslot for running the experiment, such as 1 hour, 1 day, 1 week... At the moment, estimating the running time seems challenging.

Are there some rules of thumb or some ways to estimate what should be the value for permutation count and number of examples to use to get good results? On other hand, are there some ways to estimate or make quick experiments to see how long it will take to complete? Or, are there ways to see the progress of PFI during the run?

Potential solutions

• rules of thumb on how to set number of permutations and number of examples
• guidelines on how to estimate how increasing number of examples and permutations affect running time
• PMI should report the progress

[torronen]

Could someone give me some insights on what PFI is expected to do based on the parameters? I have a model which took about 2 minutes to train. I set permutation count to 30 and numberOfExamplesToUse is 20% of the original dataset row count. So far it has been running about 8 hours. I think I may have mistunderstood what the parameters do. Any links or insights would be highly appreciated. On another machine, I have permutation count = 3 and number of examples to use let be default. It is a big model / more expensive to train, but has been running over 48 hours.

Column count is high, over 1000 in both models, which is probably the reason for the low performance. How could I estimate the impact of column count? Especially, can I somehow estimate if the runs would complete within hours, days or weeks?

[michaelgsharp]

@justinormont do you know of any good ways to estimate PFI time?

[torronen]

30 permutations with 20% row count completed in about 36-48 hours. This had a nice even CPU consumption on 6 cores.

3 permutations with full example size is still running. The models are not the same, but the dataset is the very similar. This one has a "spiky" CPU consumption on 32 cores.

Based on this first example, would it make sense to have a high permutation count but with low example count (such as 3-5% on big data set)?

[torronen]

The run with 3 permutations using all observations finished now about 1.5 weeks later on a 32-core EPYC (vs. 6-core i7 in a less than 2 days with x10 permutations but 1/5 observations). It seems observation count heavily affects the time required. The results are a bit unintuitive so I am concerned if it is correct. I could not find guidelines on what would be sensible values for permutations or observation count. Any tips for it?