OVA-PriorPredictor fails

[justinormont]

When placed in a One-vs-All (OVA), PriorPredictor throws a "Nullable object must have a value" error.

Error:

Unhandled exception. System.InvalidOperationException: Nullable object must have a value.
   at System.Nullable`1.get_Value()
   at Microsoft.ML.Data.SchemaBindablePredictorWrapperBase.SingleValueRowMapper.GetInputColumnRoles()+MoveNext()
   at Microsoft.ML.Data.RoleMappedSchema.MapFromNames(DataViewSchema schema, IEnumerable`1 roles, Boolean opt)
   at Microsoft.ML.Data.RoleMappedSchema..ctor(DataViewSchema schema, IEnumerable`1 roles, Boolean opt)
   at Microsoft.ML.Data.PredictedLabelScorerBase.BindingsImpl.ApplyToSchema(DataViewSchema input, ISchemaBindableMapper bindable, IHostEnvironment env)
   at Microsoft.ML.Data.PredictedLabelScorerBase..ctor(IHostEnvironment env, PredictedLabelScorerBase transform, IDataView newSource, String registrationName)
   at Microsoft.ML.Data.MulticlassClassificationScorer..ctor(IHostEnvironment env, MulticlassClassificationScorer transform, IDataView newSource)
   at Microsoft.ML.Data.MulticlassClassificationScorer.ApplyToDataCore(IHostEnvironment env, IDataView newSource)
   at Microsoft.ML.Data.RowToRowScorerBase.ApplyToData(IHostEnvironment env, IDataView newSource)
   at Microsoft.ML.Data.PredictionTransformerBase`1.Transform(IDataView input)
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
   at Program.Main()

PriorPredictor does work properly when not in an OVA (aka, used as a normal binary trainer).

Repro:

See fiddle: https://dotnetfiddle.net/bs5hiw

Fiddle demonstrates:

• Binary PriorPredictor is successful
• OVA-AveragedPerceptron is successful
• OVA-PriorPredictor fails

Relevant code:

var baselineMulticlassPipeline = mlContext.Transforms.Conversion.MapValueToKey("Label", "Label")
			.Append(mlContext.MulticlassClassification.Trainers.OneVersusAll(mlContext.BinaryClassification.Trainers.Prior(labelColumnName: "Label"), labelColumnName: "Label"));
		var multiclassModel = baselineMulticlassPipeline.Fit(split.TrainSet);

Cause:
PriorPredictor differs from other trainers in part as it doesn't take in a Features column (and only looks at the Label column). My first guess would be when wrapping the One-vs-All in an Estimators API, we missed the part that the Features column won't exist for the PriorPredictor.

[justinormont]

Why OVA-PriorPredictor:
The PriorPredictor is good at establishing baseline metrics from which you can measure improvement. The trainer always guesses the largest class in your dataset. The PriorPredictor is trained on your training dataset and evaluated on your validation/test dataset(s) (or cross-validation), giving you a measure of how well a NOOP trainer performs, including factors like distribution drift and sampling/splitting variability.

Docs: https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.standardtrainerscatalog.prior

Nit bug -- "models" is misspelt here:

machinelearning/src/Microsoft.ML.StandardTrainers/StandardTrainersCatalog.cs

Lines 857 to 858 in 5dbfd8a

	/// This trainer uses the proportion of a label in the training set as the probability of that label.
	/// This trainer is often used as a baseline for other more sophisticated mdels.