Description
Summary
When using IAM roles on EC2 instances, the DataSink interface does not try to use given IAM role access for the instance to access S3 buckets. It goes straight to an anonymous connection. However, there is no indication in the environment that the role is present, so it can only be accessed by trying to connect to the bucket.
It is a scenario that I had to deal with when running workflows on AWS Batch, by change the Docker image / nipype code manually (yikes!)
Actual behavior
If AWS Access Key / Secret are not defined via a file or environment variables, it tries to connect anonymously.
Expected behavior
If AWS Access Key / Secret are not defined via a file or environment variables, it should give a try to IAM Roles before setting the anonymous connection.
How to replicate the behavior
https://gist.github.com/anibalsolon/f7586a85942792432e98470c512588b9
After creating this, you need to create an EC2 instance w/ this profile, and run a workflow w/ DataSink without informing Access Key & Secret. It will try to connect anonymously.
Script/Workflow details
Just a DataSink node for S3 usage.
Platform details:
{
'pkg_path': '/home/anibalsolon/Documents/nipype/anibalsolon/nipype',
'commit_source': 'repository',
'commit_hash': 'ad93735',
'nipype_version': '1.1.0-dev+gad93735',
'sys_version': '3.6.4 |Anaconda custom (64-bit)| (default, Jan 16 2018, 18:10:19) \n[GCC 7.2.0]',
'sys_executable': '/opt/anaconda3/bin/python',
'sys_platform': 'linux',
'numpy_version': '1.14.0',
'scipy_version': '1.0.0',
'networkx_version': '2.1',
'nibabel_version': '2.2.1',
'traits_version': '4.6.0'
}
1.1.0-dev+gad93735
Execution environment
Choose one
- Container [Tag: ce73e99f008e / py36]
- My python environment outside container