Use Instance profile credential

[sumitag]

Is it possible to use temporary credential retrieved from instance profile/role?

[dekobon]

Would this be the aws_session_token token that is often used with SSO on AWS accounts? Can you provide some more details about what type of temporary credential you are thinking of?

[vli425]

A very important enhancement request actually. It is considered to be very insecure to save a permanent AWS_ACCESS_KEY and AWS_SECRET_KEY somewhere. It's even forbidden in a lot of enterprise environments. Instead roles should be used to allow AWS components to access services they need.

So let's assume this nginx-s3-gateway would run on a AWS EC2 instance, with the right role applied to the instances it would be allowed to access the S3 bucket with temporary keys which can be retrieved from the EC2 metadata, as described here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

That would be a huge security improvement for the nginx-s3-gateway, so I clearly vote up for this, new feature request.

[rinrinne]

"Retrieve security credentials from EC2 instance metadata or ECS task metadata endpoint" ?

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html

[sumitag]

"Retrieve security credentials from EC2 instance metadata or ECS task metadata endpoint" ?

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html

Depending on how you are running it, if nginx is running on ec2 then instance if ecs then task.

[ksandrmatveyev]

Hi,
@dekobon , are there any news or plans about that?

[ajschmidt8]

This feature would also be hugely beneficial to my team.

[dekobon]

I would like some help from the community in designing this feature. I've got the following questions:

Do the security credentials need to be refreshed at a periodic interval? If so, it will be difficult to implement the feature purely within nginx.

I can imagine a few implementations:

• If you aren't running Docker and using the s3 gateway with a systemd service, you could have systemd get the credentials periodically and restart nginx.
• If you are running Docker, you could use nginx-wrapper to wrap the executing process, get credentials periodically and restart NGINX.
• Maybe, this is a big maybe, a subrequest could be done when a njs variable containing a timestamp is exceeded which then resets to security credentials. This would result in increased latency for the request that would trigger the credentials update.

Does anyone have other ideas about how to implement this?

[ajschmidt8]

• Maybe, this is a big maybe, a subrequest could be done when a njs variable containing a timestamp is exceeded which then resets to security credentials. This would result in increased latency for the request that would trigger the credentials update.

Appreciate your response! I had looked into implementing the IAM role feature, and this is the path that I had initially considered. Though the other approaches seem interesting as well.

I'll highlight some of the information that I've found in my research so far.

• Containers can be run in two environments on AWS: Elastic Container Service (ECS, my team uses this) or Elastic Cloud Compute (EC2)
• The endpoints and procedures for retrieving temporary credentials from each of the above environments is slightly different
  • ECS requires a single HTTP request to retrieve credentials: curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI (link)
  • EC2 requires two HTTP requests to retrieve credentials: TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access (link)
• When using temporary credentials, a Token is issued in addition to an AccessKeyId and SecretAccessKey. The Token must be included in all API requests that use temporary credentials (this differs from the long-lived credentials, which don't have a Token to be included in API requests)
• The credentials that are issued are temporary, though I think you can set how long you'd like them to last. Which may be helpful if you're setting up some sort of background job to "get the credentials periodically" like you mentioned

Hopefully this is helpful. I'll continue to explore this on my own as time permits and I'm happy to help do any research that might get this feature implemented quickly!

[ajschmidt8]

EC2 requires two HTTP requests to retrieve credentials: TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access (link)

I don't know if this is accurate. If you click on the IMDSv1 tab from that page, it shows a single HTTP request that looks like it can be used to get the temporary credentials for EC2.

[FlorianSW]

I'm a complete newbie to how this project works :) However, I like it 👍
I just wanted to answer the same information as @ajschmidt8 posted already, so I will just extend on that :P

The expiration of credentials for a role of an ECS task running on the fargate compute type is, by default and currently, 6 hours, iirc. That means, we could hold a copy of these credentials for quite a long time actually. The response from the endpoints mentioned by @ajschmidt8 are also containing the date and time when these credentials will expire. At least for EC2 (I assume it's the same for ECS), credentials are automatically renewed by the endpoints at least 5 minutes before the current ones expire.

While looking at the Compatibility of the njs module, I came to a quick idea of how this could be implemented:

• Right now, the example template for a config sets a variable for the s3 auth during startup (?): https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/etc/nginx/templates/default.conf.template#L22
• We could move this part to the place where this auth header is set: https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/etc/nginx/templates/default.conf.template#L80 and https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/etc/nginx/templates/default.conf.template#L114
• The s3auth method would need to be changed to:
  • Hold a "local" copy of the current credentials in a kind of global state (there is the question: I assume that the script is loaded once and the global state of the script file, at least, is hold in memory and not re-created every time a method of the script is called?)
  • If there is no local copy of credentials, do the token fetch procedure (as outlined by @ajschmidt8)
  • Use a little setTimeout helper to "schedule" a refresh of the credentials (by doing the same fetch-procedure again) 1-5 minutes before the credentials expire

By doing so, nginx fetches the credentials asynchronously to actual requests (except for the first one, but even that could be worked around by adding a refreshCredentials or so method which is called when the server starts?). Preventing any overhead to the webrequests by the credential flow.

What do you think? Would that be possible, or did I miss an important details? :)

[FlorianSW]

EC2 requires two HTTP requests to retrieve credentials: TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access (link)

I don't know if this is accurate. If you click on the IMDSv1 tab from that page, it shows a single HTTP request that looks like it can be used to get the temporary credentials for EC2.

While IMDSv1 is still possible to use, one can disable v1 on a per-instance configuration. Also, v2 protects against some possible attacks which were possible with v1 (https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/). Because of that, implementing a new feature, which requires v1, might be difficult to justify. There might even be environments, where the v1 must be disabled and not be used for security or compliance reasons.

I also think that, with an async approach, using v1 doesn't really give a big benefit anymore (compared to on a per-request basis).

[FlorianSW]

I spiked something together :D It's not nearly in a state to be in a PR but maybe it is already enough to get a discussion started about my proposed solution: https://github.com/nginxinc/nginx-s3-gateway/compare/master...FlorianSW:issue/8?expand=1 (I hope you can see this diff)

• It is completely untested in an ECS or EC2 environment (I need to check out somehow where I get an ECS cluster from, I'm, however, mostly sure I will not be able to check that on an EC2 instance unfortunately. However, once the ECS thingie works, we can still try to find someone to check the EC2 version, they're mostly the same, except what API endpoints are used)
• This code for now only handles the initial credentials fetching, not the refreshing, however, adding a setTimeout should be easy enough, once we know the current logic works, kind of.
• The code might benefit from a bit of refactoring :D

I'll explore a bit more in the next days or weeks whenever I find a bit of time. However, any feedback is welcome :) (especially int he njs area, as this is completely new to me).