File transfer

Transferring files and folders with scp

To copy a single file to/from the cluster, we can use scp:

[local]$ scp /path/to/local/file.txt userXX@sfuschool.c3.ca:/path/on/remote/computer
[local]$ scp local-file.txt userXX@sfuschool.c3.ca:   # will put into your remote home
[local]$ scp userXX@sfuschool.c3.ca:/path/on/remote/computer/file.txt /path/to/local/

To recursively copy a directory, we just add the -r (recursive) flag:

[local]$ scp -r some-local-folder/ userXX@sfuschool.c3.ca:target-directory/

You can also use wildcards to transfer multiple files:

[local]$ scp userXX@sfuschool.c3.ca:start*.sh .

With MobaXterm in Windows, you can actually copy files by dragging them between your desktop and the left pane when you are logged into the cluster (no need to type any commands), or you can click the download/upload buttons.

Note

You can watch a video for this topic after the workshop.

Transferring files interactively with sftp

scp is useful, but what if we don’t know the exact location of what we want to transfer? Or perhaps we’re simply not sure which files we want to transfer yet. sftp is an interactive way of downloading and uploading files. Let’s connect to a cluster with sftp:

[local]$ sftp userXX@sfuschool.c3.ca

This will start what appears to be a shell with the prompt sftp>. However, we only have access to a limited number of commands. We can see which commands are available with help:

sftp> help
Available commands:
bye                                Quit sftp
cd path                            Change remote directory to 'path'
chgrp grp path                     Change group of file 'path' to 'grp'
chmod mode path                    Change permissions of file 'path' to 'mode'
chown own path                     Change owner of file 'path' to 'own'
df [-hi] [path]                    Display statistics for current directory or
                                   filesystem containing 'path'
exit                               Quit sftp
get [-afPpRr] remote [local]       Download file
reget [-fPpRr] remote [local]      Resume download file
reput [-fPpRr] [local] remote      Resume upload file
help                               Display this help text
lcd path                           Change local directory to 'path'
lls [ls-options [path]]            Display local directory listing
lmkdir path                        Create local directory
ln [-s] oldpath newpath            Link remote file (-s for symlink)
lpwd                               Print local working directory
ls [-1afhlnrSt] [path]             Display remote directory listing
...

Notice the presence of multiple commands that make mention of local and remote. We are actually browsing two filesystems at once, with two working directories!

sftp> pwd    # show our remote working directory
sftp> lpwd   # show our local working directory
sftp> ls     # show the contents of our remote directory
sftp> lls    # show the contents of our local directory
sftp> cd     # change the remote directory
sftp> lcd    # change the local directory
sftp> put localFile    # upload a file
sftp> get remoteFile   # download a file

And we can recursively put/get files by just adding -r. Note that the directory needs to be present beforehand:

sftp> mkdir content
sftp> put -r content/

To quit, type exit or bye.

Exercise

Using one of the above methods, try transferring files to and from the cluster. For example, you can download bfiles.tar.gz to your laptop. Which method do you like best?

Note on Windows line endings
  • When you transfer files to from a Windows system to a Unix system (Mac, Linux, BSD, Solaris, etc.) this can cause problems. Windows encodes its files slightly different than Unix, and adds an extra character to every line. On a Unix system, every line in a file ends with a \n (newline), whereas on Windows every line in a file ends with a \r\n (carriage return + newline). This causes problems sometimes.
  • In some bash implementations, you can identify if a file with Windows line endings with cat -A filename. A file with Windows line endings will have ^M$ at the end of every line. A file with Unix line endings will have $ at the end of a line.
  • Though most modern programming languages and software handle this correctly, in some rare instances you may run into an issue. The solution is to convert a file from Windows to Unix encoding with the command dos2unix filename. Conversely, to convert back to Windows format, you can run unix2dos filename.
Note on syncing

There also a command rsync for synching two directories. It is super useful, especially for work in progress. For example, you can use it the download all the latest PNG images from your working directory on the cluster.

Copy a file to/from the training cluster using either scp or sftp.

Bring up the manual page on rsync, then use it to synchronize a directory from the training cluster.

Write a command to scp multiple files with a single command so that scp asks you for password (and possible MFA) only once for all files, not once per file. Feel free to google this problem. If you have an Alliance account, try copying a set of files from the training cluster to Cedar (you will run into a problem):

for i in {1..5}; do
    echo ${RANDOM}${RANDOM}${RANDOM} > a$i.txt
done
scp a*.txt username@cedar.alliancecan.ca:scratch

How many times does it ask for a password?

Let’s discuss remote system profiles in ~/.ssh/config. Consider this:

Host <name>
    HostName sfuschool.c3.ca
    User userXX
  • Note that in Windows you can create .ssh/config in C:\Users\username. If referencing this folder from Windows Subsystem for Linux, make sure to run chmod 600 ~\.ssh\* after creating the config file.
  • Another useful setting is IdentityFile private_ssh_key coupled with IdentitiesOnly yes (i.e. use only the provided key).