Unable to run train with query data

[AlanTeoYueYang]

Report

Hi, I am trying to run a similar workflow as shown on https://multimil.readthedocs.io/en/latest/notebooks/paired_integration_cite-seq.html. I have successfully tested it on the tutorial data, but when I try it on multiple datasets, it only worked on one of them. For most of them, the error is ValueError: optimizer got an empty parameter list. The training set runs normally but the when creating a new query dataset, it doesn't seem to work. The script that I am using is shown below

query = adata[adata.obs['split_by_set']=='test'].copy()
adata = adata[adata.obs['split_by_set']=='train'].copy()

mtm.model.MultiVAE.setup_anndata(
adata,
categorical_covariate_keys=['replicate'],
rna_indices_end=rna_indices_end
)

vae = mtm.model.MultiVAE(
adata,
losses=["nb"]
)
vae.train()

new_vae = mtm.model.MultiVAE.load_query_data(query, reference_model=vae)
### ERROR IS FROM THIS STEP ###
new_vae.train(weight_decay=0)

An example error log is shown below

new_vae.train(weight_decay=0)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/multimil/model/_multivae.py", line 364, in train
  return runner()
         ^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/scvi/train/_trainrunner.py", line 82, in __call__
  self.trainer.fit(self.training_plan, self.data_splitter)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/scvi/train/_trainer.py", line 193, in fit
  super().fit(*args, **kwargs)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
  call._call_and_handle_interrupt(
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
  return trainer_fn(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
  self._run(model, ckpt_path=self.ckpt_path)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 1093, in _run
  self.strategy.setup(self)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/strategies/single_device.py", line 74, in setup
  super().setup(trainer)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/strategies/strategy.py", line 154, in setup
  self.setup_optimizers(trainer)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/strategies/strategy.py", line 142, in setup_optimizers
  self.optimizers, self.lr_scheduler_configs, self.optimizer_frequencies = _init_optimizers_and_lr_schedulers(
                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/core/optimizer.py", line 180, in _init_optimizers_and_lr_schedulers
  optim_conf = model.trainer._call_lightning_module_hook("configure_optimizers", pl_module=model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 1356, in _call_lightning_module_hook
  output = fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/scvi/train/_trainingplans.py", line 575, in configure_optimizers
  optimizer1 = self.get_optimizer_creator()(params1)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/scvi/train/_trainingplans.py", line 362, in <lambda>
  return lambda params: optimizer_cls(
                        ^^^^^^^^^^^^^^
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/torch/optim/adamw.py", line 72, in __init__
  super().__init__(params, defaults)
File "~/miniconda3/envs/clock_python_env/lib/python3.12/site-packages/torch/optim/optimizer.py", line 362, in __init__
  raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list

Do you have any solutions?

Version information

---

anndata 0.10.9
multimil 0.2.0
muon 0.1.6
numpy 1.26.4
pandas 1.5.3
scanpy 1.10.2
scipy 1.14.1
scvi 0.20.3
session_info 1.0.0
torch 2.4.1+cu124

PIL 10.4.0
absl NA
attr 24.2.0
certifi 2024.08.30
charset_normalizer 3.3.2
chex 0.1.86
contextlib2 NA
cycler 0.12.1
cython_runtime NA
dateutil 2.9.0.post0
distutils 3.12.5
docrep 0.3.2
etils 1.9.4
filelock 3.13.1
flax 0.9.0
fsspec 2024.2.0
h5py 3.11.0
idna 3.8
igraph 0.11.6
importlib_resources NA
jaraco NA
jax 0.4.31
jaxlib 0.4.31
joblib 1.4.2
kiwisolver 1.4.7
legacy_api_wrap NA
leidenalg 0.10.2
lightning_fabric 1.9.5
lightning_utilities 0.11.7
llvmlite 0.43.0
matplotlib 3.9.2
ml_collections NA
ml_dtypes 0.4.0
more_itertools 10.3.0
mpl_toolkits NA
mpmath 1.3.0
msgpack 1.0.8
mudata 0.3.1
multipledispatch 0.6.0
natsort 8.4.0
numba 0.60.0
numpyro 0.15.2
opt_einsum v3.3.0
optax 0.2.3
packaging 24.1
patsy 0.5.6
pkg_resources NA
platformdirs 4.2.2
pygments 2.18.0
pynndescent 0.5.13
pyparsing 3.1.4
pyro 1.9.1
pytorch_lightning 1.9.5
pytz 2024.1
requests 2.32.3
rich NA
seaborn 0.13.2
setuptools 73.0.1
six 1.16.0
sklearn 1.5.1
socks 1.7.1
statsmodels 0.14.2
sympy 1.12
texttable 1.7.0
threadpoolctl 3.5.0
toolz 0.12.1
torchgen NA
torchmetrics 1.4.1
tqdm 4.66.5
triton 3.0.0
typing_extensions NA
umap 0.5.6
urllib3 2.2.2
yaml 6.0.2

Python 3.12.5 | packaged by conda-forge | (main, Aug 8 2024, 18:36:51) [GCC 12.4.0]
Linux-5.14.0-70.30.1.el9_0.x86_64-x86_64-with-glibc2.34

Session information updated at 2024-09-22 11:20