Can't control arguments in fit_gpytorch_mll under the hood. Getting ABNORMAL_TERMINATION_IN_LNSRCH warning

[Jgmedina95]

During model optimization im getting the following warning:
scipy_minimizeterminated with status 3, displaying original message fromscipy.optimize.minimize`: ABNORMAL_TERMINATION_IN_LNSRCH. And recently, it tried several times after raising the error.

The traceback of the error falls here:

    [288]     mll = mll_class(likelihood=m.likelihood, model=m, **mll_options)
--> [289]     fit_gpytorch_mll(mll)
    [290] else:
    [291]     raise NotImplementedError(
    [292]        f"Model of type {m.__class__.__name__} is currently not supported."
    [293]

    [102] if optimizer is not None:  # defer to per-method defaults
    [103]    kwargs["optimizer"] = optimizer
--> [105]return FitGPyTorchMLL(
    [106]     mll,
    [107]    type(mll.likelihood),
    [108]    type(mll.model),
    [109]   closure=closure,
    [110]    closure_kwargs=closure_kwargs,
    [111]    optimizer_kwargs=optimizer_kwargs,
    [112]    **kwargs,
    [113]
File  in Dispatcher.__call__(self, *args, **kwargs)
     [91] func = self.__getitem__(types=types)
     [92] try:
---> [93]     return func(*args, **kwargs)
     [94] except MDNotImplementedError:
     [95]   # Traverses registered methods in order, yields whenever a match is found
     [96]    funcs = self.dispatch_iter(*types)

File  in _fit_fallback(mll, _, __, closure, optimizer, closure_kwargs, optimizer_kwargs, max_attempts, warning_handler, caught_exception_types, **ignore)
    [280]if debug.off():
    [281]  msg = msg + " For more information, try enabling botorch.settings.debug mode."
--> [283]raise ModelFittingError(msg)

ModelFittingError: All attempts to fit the model have failed. For more information, try enabling botorch.settings.debug mode.

After some reading, I couldn't figure out why the error occurred. Suggestions online say it is most likely numerical stability + model fitting the data poorly. S

Even then I tried to see how to force extra retries and here is where I noticed that even that fit_gpytorch_mll allows for a max_attempts argument. There is no way to include it through Ax. So I'm playing with different choices of mll to see if it avoids this issue.

My question would be, if that error has come in the past, what suggestions do you have to go around it? It is worth noting that this only occurred after some rounds in the loop; the first rounds, things went smoothly. And I would really dislike changing model choices mid-optimization.

Thanks for your time, and please lmk if you would like a notebook to replicate the error :)

[bernardbeckerman]

@Jgmedina95 thanks for posting! Let me find someone internally that can answer your question and get back to you. In the mean time can you provide a bit more information as to when this is happening, e.g., on what kind of problem is this occurring? Please provide sufficient detail so we can recreate the problem locally if possible.