Measurement-only variable

[JamesBagley]

Hi all

I'm trying to add variables for elements like ambient temperature / humidity that might have day to day variation but cannot be directly controlled for an experiment. I think I've found a way to do that for NumericalDiscreteParameters via DiscreteCustomConstraint. It has the expected behavior of loading in my prior data with varying temperature values without error and all recommended experiments have the suggested lab temp of 22

e.g.

Lab_temperature = NumericalDiscreteParameter(
    name=f"Lab_temperature",
    values=[20, 21, 22, 23, 24, 25, 26],
    tolerance=0.49
)

def temp_filter(df: pd.DataFrame) -> pd.Series:  # this signature is required
    """
    we don't control this, fix suggestions to 22
    """
    mask_good = df["Lab_temperature"]==22
    return mask_good

Lab_temperature_constraint = DiscreteCustomConstraint(
    parameters=[  # the custom function will have access to these variables
        "Lab_temperature",
    ],
    validator=temp_filter,
)

However I've tried to do the same for a categorical parameter with the same technique

Lab_practitioner = CategoricalParameter(
    'Lab_practitioner',
    ['Jack', 'Jill', 'Mary', 'Unknown', 'any']
)

def practitioner_filter(df: pd.DataFrame) -> pd.Series:  # this signature is required
    """
    We don't want to specify who should run the experiment, but maybe has value as a retrospective
    """
    mask_good = df["Lab_practitioner"]=='any'
    return mask_good

Lab_practitioner_constraint = DiscreteCustomConstraint(
    parameters=[  # the custom function will have access to these variables
        "Lab_practitioner",
    ],
    validator=practitioner_filter,
)

And while the recommendations produced do consistently have the "Lab_practitioner" parameter set to "any", on the loading of my pre-existing dataset I get the following error for every row of the dataframe:

WARNING:baybe.utils.dataframe:Input row with index 0 could not be matched to the search space. This could indicate that something went wrong.

I'm wondering if this is an intended use case and if so whether there would be a alternate way to implement something like this

[Scienfitz]

Hey @JamesBagley ,

What happens with discrete constaints including CustomDiscreteConstraint
After the searchspace is built form all parameter combinations you specified, we filter the entries according to all discrete constraints you provided. This means that even though you specify a CategoricalParameter with values=['A','B','C'], the searchspace will contain only entries with A if you also provided a constraint that marks B and C as invalid.
BTW you could also use a DiscreteExludeConstraint with a SubSelectionCondition in your case because its about simple filtering of values.

Why does that cause the error you're seeing
Here we have to make a distinction between NumericalDiscreteParameter and CategoricalParameter

For CategoricalParameter, the searchspace will only contain entries where parameter Lab_practitioner has value any. I assume your pre-existing dataset also contains entries where this parameter has a different value. This is causing the error, as upon adding data, a match between the input rows and the searchspace is performed, failing because the searchspace has been filtered by your constraint.

For NumericalDiscreteParameter the match does not have to be exact. Input rows just have to be inside the tolerance around the specified values or if there is no tolerance it will be accepted in any case. Now there is a chance that there is a bug or somehing similar if you were still allowed to input a value of 20 if your constraint should have filtered everything except 22 with a tolerance of 0.49, but your have to give some insight on the data for us to investigate that.

In the meantime, you can continue fixing your problem because

Constraints are not the right thing for your problem
You actually don't want to filter the searchspace, you simply want to restrict recommendations. In the current version this is possible

Using Metadata
For instance like

# find indices of parameter values that you dont want recommended
dont_recommend_idxs = searchspace.discrete.exp_rep['Lab_temperature'] != 22

# restrict recommendations via metadata 
searchspace.discrete.metadata.loc[dont_recommend_idxs, 'dont_recommend'] = True

I think given the filtration capabilities you have here when doing the index search makes values like Unknown or any for the practitioner variable maybe obsolete. Or you simply always set dont_recommend for all entries except any to True. Then you could read in from any practitioner, but the algo wouldnt care about the practitioner when recommending.

This is somewhat cumbersome, so we plan to generalize

The active_values formalism
This formalism already exists for the TaskParameter. values defines all values that should be possible to read in via add_measurements while active_values (must be a subset of values) controls the values that are recommended. This is not yet generalized to all discrete parameters, but will be. In the meantime please use the metadata.

Exogeneous parameters
What your trying to do here is called exogeneous parameters in our formalism, values that an be measured but not controlled. We will have either a special parameter or a special parameter flag for that in the future. With that you would neither have to use active_values nor metadata, but would simply provide the exogeneous parameter values of the current situation at point of calling .recommend. Be sure to watch out for when we release this.

[JamesBagley]

Thanks for the response!

I just tried it out with your suggestion and it seems to be working. The active_values formalism you're describing sounds like it would be very intuitive solution, I was just double checking the docs to see if there was a way to either fix variables to a specific value or limit the search space via the recommend call.

I also went back and looked at the temperature values though and I can confirm it doesn't raise any warnings even when the temperature was set as high as 26 with the 0.49 tolerance. Increasing it to 28 causes a ValueError as it's out of the 20-26 range that I established when declaring the variable.

[AdrianSosic]

Hi @JamesBagley and @Scienfitz, I just wanted to briefly pick up on this discussion and add a few points:

1. Fully agree with @Scienfitz about the exogenous parameters -> this is what you want for things that you cannot control but still would like to track in your data / feed into the models. We absolutely have this on our bucket list and I hope we can support it soon.
2. I don't fully agree with the constraints are not the right thing for your problem statement. While exogenous parameters definitely offer the nicer approach, I don't see why going via constraints shouldn't be OK (as a workaround for the exogenous part). In fact, if everything was set up correctly in our input data matching pipeline, dropping entries from the search space should be equivalent to setting something to dont_recommend, because I argue the data matching should be performed on the parameter level and not the search space level (in fact, I'd even argue the dont_recommend metadata is technically not even needed and might either vanish in the future or will at least be hidden behind some friendly user interface – but that's a different discussion).
3. This brings me to the most important part: a quick inspection showed me that the matching part of our data parsing routine is not 100% clean, which is why @JamesBagley runs into the issue. While this does relate to the tolerance (which is planned to refactored, btw), I think there is a more deeper issue that needs to be fixed. @Scienfitz: we can discuss this in the next dev meeting. @JamesBagley: Until we've aligned internally on how to handle this, I'd recommend you stick with @Scienfitz's suggestion and use the metadata workaround for now. I'll ping you again once we have a better answer 👍🏼