Setting up Globus on Jetstream
May 22 2018
Jetstream is an XSEDE, cloud computing that provides access to preconfigured virtual machines (VM) with root access. This is an NSF funded project between Indiana University and Texas Advanced Computing Center (TACC), for more information on Jetstream go here.
Interested in using Jetstream for your research here is a tutorial on "Getting started on Jetstream".
File transfer to/from Jetstream
File transfer to a Jetstream instance can be done through several ways. For small files, such as a simple two-line text file, copy pasting the content is probably just as quick as transferring the file. When you get to larger or multiple files, such as a directory (folder) or bigger file, you can transfer it using:
- WinSCP , FileZilla (windows users) or
CyberDuck(windows and OS).
Using these commands to transferring files and managing data is also covered here.
However, when you have really large files, such as raw sequence data, it can be a bit tedious to transfer files via command line, not to mention problematic if your internet connection drops out. In these cases, Globus is a favorable alternative. Globus allows you to transfer gigabytes or terabytes of data securely and quickly.
Before you begin
Jetstream being an XSEDE resource and already has a
This is a tutorial walking through setting up
Setting up Globus endpoint Jetstream VM
These are basically the instructions for the set up of Globus on any Linux machine. However, an example on Jetstream with screen captures is provided to assist newer users.
Hopefully, you should already have an account on Globus - if not goto here, and have a Jetstream virtual machine setup, if not go here. In the following instructions, lines in red are commands, the portions in purple indicate that you need fill in the information. I have also attached screenshots of how a screen will look if the command ran successfully for the first few commands, to gain some confidence that you are doing it right.
Log into the VM (terminal in WebDesktop, Webshell or ssh- your preference). If you are new to command line, “Welcome to the Dark Side!”. You can access the WebDesktop (red) and Webshell (black) with the following links when you click on an active image in your Jetstream Atmosphere Portal. In this case, you will want to use the Web Shell.
NOTE: If the beta is still listed as such, please use that. The old version does not have support for copy and paste. If there is no longer a beta listed, it is likely that the new system has fully replaced the old and you can just use the only web shell listed.
You can also run these commands via putty (Windows), ssh command on terminal (LINUX/MAC), you can log in to your VM using your username, IP address (can be found under Jetstream instance details), your key, port 22. This can make things a bit easier if you are comfortable using terminals. For more information here.
If you are working on the Webshell, the main weirdness you will run into is using copy or paste from your computer to Webshell and vice-versa. Here is the trick, on the WebShell:
Windows (shift+ctl+alt)/Mac (shift+ctrl+cmd) -> This brings up a sidebar, with a box for use as a common clipboard.
- To paste from your computer to the VM, you will have to paste the text here. Then, exit the sidebar with shift+ctl+cmd, and paste into the shell with shift+ctl+alt/shift+ctrl+v(mac).
- To paste from the VM to your computer, highlight the text, bring up the sidebar, and make sure the text is in the common clipboard, then paste as normal.
- To paste from the VM to the VM, you can just highlight the text and paste as normal (not going across computers).
Once you are in a terminal of your choosing, follow these steps:
- In your home directory, or wherever you install files, download
globusconnect using this command: wgethttps://s3.amazonaws.com/connect.globusonline.org/linux/stable/globusconnectpersonal-latest.tgz
- Unzip the
globusconnect file downloaded:
- Make a new ssh-key for the instance:
rsa-b 2048 -f js
This command will create a public and private key at current working directory under the name “js”. Type in a passphrase here- acts as 2 step verification process.
Once done, you will see a pattern as shown below.
- Move the “js.pub” and “js” to ./ssh
mv js* ~/.ssh/
chmod 600 js*# this command changes the permissions to the key pair.
- Add a new key (public) to
Click on Account
- In accounts, click on “ manage SSH and X.509 keys.”
IF YOU DO NOT SEE THE ABOVE OPTION: Look at the linked identities under Account. If you do not see an account ending in “globusid.org”, you will need to get one. A
globusidis required to add keys. You are able to log in with a gmail/institutional email, and may not have a globusid.
globusid, follow this link.
Once you have signed up, you should link your account to your main account (listed under globus.org, Accounts).
The option “ manage SSH and X.509 keys” should show up now.
- Then select “add a new key”
- Fill in the information and copy paste public key to the body
- Click on “Add Key”
- Back to Jetstream instance, in the terminal go to
globusconnectdirectory. In our case, we installed in the home directory. Also make note the version may have changed since the blog post.
- Log into globusonline.org with your ssh key from your instance (you need to use your username):
ssh -i ~/.ssh/js email@example.com
- Now setup endpoint on globusonline.org for your instance
endpoint-add -n name_of_the_endpoint --
This command will give you an Endpoint ID and Setup ID for the instance, which you WILL NEED in the next set of commands. Copy paste them into a text editor if needed. Sign out of the ssh with "exit", then in the
./globusconnect -setup Setup_ID
./globusconnect -start &
ssh -i ~/.ssh/js firstname.lastname@example.org endpoint-activate Endpoint_ID
Globus connect endpoint is now setup and files can be transferred to and from Jetstream to the local computer or other places with Globus endpoints already set up. (Ex: /N/dc2/scratch/username- if you have access to IU cluster)
Note: If you are looking to transfer to a volume attached to the VM directly, follow the Step 8 here instead.
- Verification step to see if the
globusconnect endpoint has been setup
ssh -i ~/.ssh/js email@example.com endpoint-search --scope my-endpoints
- Transfer files using
globusconnect online https://www.globus.org/app/transfer
- To stop
globusconnectconnection in instance, type
DONE! You are all set!
I have logged in to
Here is the trick you need to login to
- you need to have your google account (firstname.lastname@example.org),
xsedeaccount (email@example.com), IU ID (firstname.lastname@example.org) or other accounts all listed under accounts.
- Now logout and log back in using your Google credentials
- Search for the endpoint IU#dc2 and you should be able to connect to this endpoint.
If you have any questions mail us at email@example.com