Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XML compression #27022

Merged
merged 42 commits into from
Apr 21, 2020
Merged

XML compression #27022

merged 42 commits into from
Apr 21, 2020

Conversation

satyaakam
Copy link
Contributor

https://dimagi-dev.atlassian.net/browse/IIO-742

SUMMARY

Changes in BlobDB API were made as per the suggestions on #26813

FEATURE FLAG

ICDS-CAS

PRODUCT DESCRIPTION

None

corehq/blobs/mixin.py Outdated Show resolved Hide resolved
corehq/blobs/s3db.py Show resolved Hide resolved
corehq/blobs/s3db.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
Copy link
Contributor

@snopoke snopoke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a quick pass and left some comments. Overall looks like a good start.

I don't think you meant to commit these:
image

corehq/blobs/s3db.py Outdated Show resolved Hide resolved
corehq/blobs/interface.py Outdated Show resolved Hide resolved
corehq/blobs/interface.py Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/s3db.py Outdated Show resolved Hide resolved
corehq/form_processor/models.py Outdated Show resolved Hide resolved
corehq/blobs/s3db.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/s3db.py Outdated Show resolved Hide resolved
corehq/blobs/mixin.py Outdated Show resolved Hide resolved
corehq/blobs/s3db.py Outdated Show resolved Hide resolved
corehq/blobs/util.py Outdated Show resolved Hide resolved
corehq/blobs/tests/test_util.py Show resolved Hide resolved
corehq/blobs/exceptions.py Outdated Show resolved Hide resolved
corehq/blobs/exceptions.py Outdated Show resolved Hide resolved
corehq/blobs/tests/test_util.py Show resolved Hide resolved
corehq/blobs/tests/test_util.py Outdated Show resolved Hide resolved
corehq/blobs/exceptions.py Outdated Show resolved Hide resolved
corehq/blobs/tests/test_util.py Outdated Show resolved Hide resolved
corehq/blobs/export.py Outdated Show resolved Hide resolved
@@ -31,6 +32,8 @@ def new(self, **blob_meta_args):
"keyword arguments are incompatible with `meta` argument")
return blob_meta_args["meta"]
timeout = blob_meta_args.pop("timeout", None)
if blob_meta_args.get('type_code') == CODES.form_xml:
blob_meta_args['compressed_length'] = -1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really like this. It does got overwritten during later but still feels off

corehq/blobs/interface.py Outdated Show resolved Hide resolved
corehq/blobs/metadata.py Show resolved Hide resolved
corehq/blobs/s3db.py Show resolved Hide resolved
@snopoke snopoke added the product/invisible Change has no end-user visible impact label Apr 20, 2020
snopoke and others added 5 commits April 20, 2020 16:58
Make the error more generic so it can be used in other places if need be. Currently this error is never expected to occur unless there is a bug.
Both GzipStream and GzipStream.Buffer had a `_buf` attribute. The class composition made it easy to confuse the two.
Copy link
Contributor

@millerdev millerdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed some bugs and created a new PR against the original branch in @satyaakam's fork: satyaakam#1

@snopoke I'm not sure how you were able to push directly to this PR since it's originating from a fork. I'd be interested in learning how you did that.

Gzip stream fixes for XML compression
@snopoke
Copy link
Contributor

snopoke commented Apr 21, 2020

I'm not sure how you were able to push directly to this PR since it's originating from a fork. I'd be interested in learning how you did that.

I got given write access to the forked repo

@snopoke
Copy link
Contributor

snopoke commented Apr 21, 2020

@millerdev build is passing but will let you do a final review before merging

@snopoke snopoke merged commit 3465254 into dimagi:master Apr 21, 2020
@snopoke snopoke deleted the compress-xml branch April 21, 2020 13:42
@shubham1g5
Copy link
Contributor

Flagging that this PR incorrectly updated the commcare-translations submodule reverting it back to an older 2.47 version. @orangejenny has made the required correction on master pushing it back to the latest commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
product/invisible Change has no end-user visible impact
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants