AWS Instance Profile & IAM Role: an Unholy Marriage

TL;DR

The complexities between AWS IAM roles and instance profiles can result in unexpected behaviors in EC2 instances. This article provides an in-depth look into these intricacies, highlighting the role of security tokens, cross account operations, and offers actionable insights for effective management.

Exploring AWS IAM role & instance profile quirks

Navigating the realms of identity and access management (IAM) roles and instance profiles is essential when you're trying to grant EC2 instances access to AWS resources. When you delve into the subject, certain behaviors, which might seem unexpected at first, emerge, emphasizing the importance of AWS IAM user guide references.

Key AWS concepts

IAM role: 
IAM roles act as a security token or a virtual "key," offering specific permissions to access AWS resources. They become particularly pivotal during cross account scenarios where users or roles from one AWS account need access to resources in another. It’s an effective method to provide EC2 instances access without resorting to hardcoding sensitive information. Moreover, the ARN of the role is vital to understand the scope and specifics of its access.

Instance profile: 
Beyond being a cost center, an instance profile serves as a bridge for IAM roles. When you add a tag or an iam tag, especially when tagging IAM resources, you streamline and refine permissions. Launching an EC2 instance with a properly tagged profile ensures more precise AWS resource management.

Instance metadata service (IMDS):
IMDS, often laden with environment variables, is a service present on every EC2 instance. This reservoir of information offers insights into the instance's network settings, attached role credentials, and other essential metadata, facilitating processes like the action sts assumerole. Such environment variables help applications retrieve information without extra configurations.

Looking to optimize your AWS security hygiene? Explore the benefits

Observations and experiments

During my in-depth testing, I utilized several CLI commands to get the hands-on experience. Taking the initiative, I attached a role named 'RedRole' to an EC2 instance. IAM users and roles come into play here: if you're using the AWS console, it simplifies the creation of the instance profile and typically names it similar to the role. The IAM user guide provides deeper insights into such operations. Intriguingly, 'RedRole', when a condition key is evaluated, comes equipped with permissions allowing it to write to S3 and even spawn an IAM user. This role seamlessly attached to the 'MyInPr' instance profile, solidifying its association with the designated EC2 instance.

Figure 1: Attaching a role to the AWS instance profile

As the application running on the instance, or the CLI, attempts to retrieve the security token bestowed by the role, the request targets the IMDS. For our narrative's sake, these credentials are 'ASIA 123'. Such temporary credentials, typified by the 'ASIA' prefix, are transient, holding validity for just six hours. Post this, IMDS replenishes them, ensuring there's no role session expiration.

A twist emerges when the role detaches from the instance profile, yet the latter remains with the EC2 instance. Both application and CLI maintained their prowess to write to S3 and even craft a new IAM user. Verification affirmed that the IMDS continued dispensing 'ASIA 123'. This might slip past a hurried look in the console, but AWS, in its IAM user guide, assures this as intended behavior.

Figure 2: Role removed from AWS instance profile

Further tests, integrating 'RedRole' again with the instance profile, displayed IMDS consistency, still dispensing the original credentials. However, the real puzzle was when swapping 'RedRole' for 'BlueRole'. Within mere seconds, the AWS console mirrored 'BlueRole' as the newly affiliated role with the EC2 instance. But the CLI took a staggering 54 minutes to confirm the transition by showcasing the new credentials. This latency, wherein the previous credentials remained functional, was enlightening.

Figure 3: Original role added back to the AWS instance profile

A subsequent test had 'YellowRole' replaced by 'OrangeRole'. Rebooting the instance and regaining CLI access disclosed that 'YellowRole' credentials persisted. If one aims to make the EC2 relinquish the use of original credentials, they'd have to enact drastic measures: eradicating the role, revoking its sessions spanning up to six hours, or bestowing a new instance profile and role onto the EC2 instance.

Figure 4: New role added to AWS instance profile

Lastly, my tests spotlighted the persistence of instance profiles, even post role deletion via the AWS console. While IAM autonomously molds instance profiles when birthing new roles, no equivalent action ensues upon role cessation. Persistently, I attached a previously deleted role profile to an EC2 instance and further infused a random role using CLI commands. This underscored a fundamental trait: instance profiles, being versatile, can accommodate any role, even if their nomenclature diverges.

Actionable takeaways

• AWS stated what I’ve reported as “expected behavior”
• A recently emptied instance profile still attached to an EC2 instance continues to allow that instance to use the role’s credentials for up to six hours
• Adding the original role back to the instance profile does not result in new credentials being acquired by the IMDS
• Swapping out the original role with a new one in the instance profile leads to the original credentials persisting until eventual consistency occurs (up to 54 minutes in my testing)
• When roles are swapped in the instance profile, the AWS console for EC2 and the CLI for the instance show different IAM roles being used for up to 54 minutes (in testing)
• To force a credential refresh being served by IMDS, you need to either delete the role, revoke its session for more than six hours, stop/start the instance (not reboot), or associate a new instance profile and role to the EC2 instance
• Even after role deletion in the AWS console, instance profiles remain and can be attached to EC2 instances, supporting various roles with potentially differing names
  
  

Commands used

• aws ec2 describe-iam-instance-profile-association
  
  
• aws iam remove-role-from-instance-profile --instance-profile-name <IP_NAME> --role-name <ROLE_NAME>
  
  
• aws iam add-role-to-instance-profile --role-name <ROLE_NAME> --instance-profile-name <IP_NAME>
  
  
• aws iam get-instance-profile --instance-profile-name <PROFILE_NAME>
  
  
• TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/<ROLE_NAME>
  
  
• aws ec2 disassociate-iam-instance-profile --association-id <ASSOCIATION_ID>

Concluding thoughts

The world of AWS IAM roles and instance profiles is dense, laden with intricacies. It’s paramount to approach it with caution and thorough knowledge. For those wanting an edge in AWS security hygiene, consider employing tools like Uptycs for AWS, ensuring swift D&R times and reduced risks.