Added scp transport for hotstart

[jasonfleming]

resolves #715 not tested yet

(edited by @wwlwpd below)

We discussed on 4/9/24, updating this PR:

• changing scp:// to ssh:// due to the upstream desire to deprecated scp
• if it looks like a file (i.e., doesn't have http://, https://, or ssh://), look for the standard HOST:/path/to/file; this can be done easily by splitting on the :, with awk -F: '{print $1}' and awk -F '{print $2}'.

These things to do will round out this support nicely. In the future, other protocols may become important (e.g., s3://, etc); so we definitely want to be able to have a clear place to support them and a consistent way to support them across the ASGS toolset.

[wwlwpd]

If you want to take this out of draft and confirm for me it was working for you last time you used it, I'd be happy to merge it if you're ready for it.

[jasonfleming]

Ok thank you. I think I made these changes as a conversation starter and never got to the testing phase. This is an important enabler, so this will be next on my readiness agenda.

[wwlwpd]

Thanks @jasonfleming - bear in mind if we just want to support it in this particular case, the I think the current implementation is fine. If we want to add scp:// support in other places, we should create a function that does parsing of the path so we're not re-implementing that all over the place. I am happy to help here.

[wwlwpd]

@jasonfleming just wondering what the state of this PR draft was

[jasonfleming]

Implemented but not tested. I just bumped the priority of the underlying issue to important non-critical

[wwlwpd]

We can discuss this after #1277 gets in. I've opted to not use a URI scheme for now, instead assuming ssh is the transport, and just using the direct syntax used by ssh, scp, and rsync (which assumes ssh), user@host:path/to/file.

more details:

Since we lean heavily on ~/.ssh/config, it's more practically, host:path/to/file. This is both easier to parse and can be used directly during backup. For restore, I just awk -F: to get host and file since the command to get the files using ssh necessarily uses the parts in a deconstructed way. See #1277 for more context to the snippets below,

BKPTGZ=${1:-$HOME/ASGS-SAVE-${PWD}.tgz
#... looks for ":" to indicate remote file path
if [[ $BKPTGZ =~ ":" ]]; then
  USE_SSH=yes
fi

...

if [ "$USE_SSH" == "yes" ]; then
  HOST=$(echo "$BKPTGZ"|awk -F: '{print $1}')
  FILE=$(echo "$BKPTGZ"|awk -F: '{print $2}')
  tar zcvf - -T ./ASGS-MANIFEST | ssh $HOST "cat > $FILE"
else         
  tar zcvf $BKPTGZ -T ./ASGS-MANIFEST
fi

The command to get a file and pipe it to tar zcvf is,

  ssh $HOST "cat $FILE" | tar zxvf -

[jasonfleming]

The usual transport for hotstart files is https ... maybe if the ":" is missing then we assume the transport is https?

[jasonfleming]

Need to include a fix for #1002 for this to be useful.