Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates for daily loading, kibo, sqlalchemy 2. #14

Merged
merged 118 commits into from
Sep 26, 2024
Merged

Conversation

weaverba137
Copy link
Member

This PR:

  • Updates SQLAlchemy code to SQLAlchemy 2 (now installed at NERSC).
  • Add a convert() method to database objects (ORM classes). Basically each table knows how to convert data tables into database objects.
  • Adds some configuration needed to load kibo.
  • Adds support for tile-based loading, which will be needed to support loading daily.
  • Adds code for patching daily exposure and tile files based on another specprod.

Note that there is some remaining work to be done:

  • Remove redundant code now that all database objects have a convert() method. This should be included in this PR.
  • Some additional work is needed to support tile-based updates (in the SQL UPDATE sense). This may or may not end up being included in this PR.

@weaverba137 weaverba137 added the wip Work In Progress label Sep 17, 2024
@weaverba137 weaverba137 self-assigned this Sep 17, 2024
@weaverba137
Copy link
Member Author

PS, some additional testing is needed to ensure that any tag resulting from this PR can support guadalupe and iron.

If kibo requires additional columns, for example, that should be a separate PR and tag.

@weaverba137
Copy link
Member Author

Also make sure to keep the tests in #12 in mind. To recap:

  • If one or more tables in a given specprod schema contain columns that are not described by the associated ORM object(s), that's OK.
  • If one or more ORM objects describe columns that are not in an associated table in a given specprod schema, that's BAD.

In other words, if we have to add columns for kibo, we have to preserve a version that does not have those columns in order to work with iron.

Finally, we may need to preserve an environment that contains SQLAlchemy 1.x in order to work with fuji.

@weaverba137
Copy link
Member Author

Note: this PR, changes the size of Exposure.goaltype and Tile.goaltype relative to iron. Make sure this change works with iron. There should not be any other column changes relative to iron in this PR. If necessary, revert those changes for the purposes of making a DR1 tag, then update the sizes for kibo.

@weaverba137
Copy link
Member Author

I tested iron with an updated tutorial notebook. Based on that, I'm going to merge and tag a version that we know still works with both iron and SQLAlchemy 2.

@weaverba137 weaverba137 merged commit 6a182d9 into main Sep 26, 2024
12 of 14 checks passed
@weaverba137 weaverba137 deleted the sqlalchemy-2 branch September 26, 2024 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wip Work In Progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant