Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snakemake output prints sensitive information with storage plugin #3087

Open
MattMonk opened this issue Sep 14, 2024 · 1 comment
Open

Snakemake output prints sensitive information with storage plugin #3087

MattMonk opened this issue Sep 14, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@MattMonk
Copy link

Snakemake version
8.20.3

Describe the bug
When using a storage plugin e.g. snakemake-storage-plugin-xrootd the user can provide a username/password or encode an authentication token in the file path. When snakemake then prints the DAG or error messages the full path, including the token/username/password whatever, is printed in the terminal exposing potentially sensitive information.

It would be great to have a way to be able to ensure that only ever a censored form of the URL is printed (but still allow for the full, uncensored URL to be passed as input/output to the shell part).

Minimal example

storage:
    provider="xrootd",
    host="eoslhcb.cern.ch",
    username="my_username"

rule test_rule:
    input: storage("root://eos/my_file.root")
    output: "test.flag"
    shell: "touch {output}"

It will correctly fail due to the file not existing and incorrect credentials:

$ snakemake -c1 test.flag -n --storage-xrootd-password="my_password"
A password has been specified -- it will be printed in plaintext when Snakemake displays the inputs/outputs of jobs! Only use this option in trusted environments.
Building DAG of jobs...
Error checking existence of root://my_username:****@eoslhcb.cern.ch:1094//eos/my_file.root on XRootD: [ERROR] Server responded with an error: [3010] Unable to give access - user access restricted - unauthorized identity used ; Permission denied
, attempt 1/3 failed - retrying in 3 seconds...
Unrecoverable error, no more retries
WorkflowError:
Failed to check existence of root://my_username:[email protected]:1094//eos/my_file.root
XRootDFatalException: Error checking existence of root://my_username:****@eoslhcb.cern.ch:1094//eos/my_file.root on XRootD: [ERROR] Server responded with an error: [3010] Unable to give access - user access restricted - unauthorized identity used ; Permission denied

where you can see (more than once) the password would be printed in plain text.

Additional context
The snakemake-storage-plugin-xrootd has this postprocess_query method which can be changed to print out a censored URL instead but, as far as I can tell, this then prevents the plugin from working correctly because the censored URL is used as the "file" rather than actual full URL.

Our particular use-case for this is running CI/CD tests where the authentication is handeled by appending a token into the URL parameters and we would prefer not to be exposing that in the CI logs if we can avoid it!

@MattMonk MattMonk added the bug Something isn't working label Sep 14, 2024
@MattMonk
Copy link
Author

I've opened a draft PR that I think would solve this: #3089

Comments/suggestions welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant