AWS - Amazon Web Services
It is possible to work with data from AWS at CCI. However due to CCI's secured environment minor changes from your normal installation and processes are neccessary.
Runtime access to S3 or any other web API end point from your HPC code running at CCI is highly discouraged. It will almost certainly cause your softwarae to be data bound while waiting for responses. Instead use a script or program to download and prestage any data you need access too on scratch in CCI's shared filesystem.
You can install the AWS CLI client at CCI using your user level permissions. Detailed instructions can be found at https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html
- Download the zip installer listed on the instructions page above to your local computer
- SCP the file to a CCI Landing pad
- Install to your barn-shared folder
./aws/install -i ~/barn-shared/aws-cli -b ~/barn-shared
- Test by running
Boto Python Library¶
In addition to the AWS CLI you can use python code and Boto to access AWS. Our existing Conda support makes installing Boto easy.
- Setup Conda
- Install Boto https://anaconda.org/anaconda/boto3 .
Example accessing s3 using Boto https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html
You will need to provide any end points in AWS you are using to CCI support so that we review them for whitelisting. In general you need to provide a DNS end point that you are in control of. For example yourbucket.s3.amazonaws.com not s3.amazonaws.com. It is possible that there are AWS services we cannot whitelist.
For S3 access you need to use a virtual-hosted-style-access url to access S3. You will need to provide the DNS name portion of this to CCI support so that we can whitelist the DNS name for the specific bucket.