Is SnapATAC2 AnnData class expected to work with scvi-tools?

[vitkl]

Hi @kaizhang

Do you expect SnapATAC2 modified AnnData class to work with scvi-tools dataloaders? cc @adamgayoso

[kaizhang]

The AnnData class has two modes in SnapATAC2 similar to the anndata package. They are:

1. backed mode: the AnnData in this mode is rewritten using Rust. However, the Python user interface remains largely the same, except that it uses polars dataframe instread of pandas dataframe. This might work out-of-box with scvi-tools. If not, it shouldn't be too hard to make it work.
2. in-memory mode: it is just the AnnData object from the anndata python package. So it should work with scvi-tools.

[vitkl]

This is great, thanks for explaining.

Do you have any idea about how mode 1 will behave with multi-GPU training where several dataloader workers need access to data?

[kaizhang]

Right now the backed mode is using the hdf5 file as the backend. hdf5 file has very limited if any, support for concurrent access. I would not advise using that in multi-GPU training. I'm working on other backends such as Zarr which is going to solve this problem. For now, try to stick with in-memory mode if possible.

[vitkl]

Good to know. Looking forward to future releases.