Settings up and structuring AWS accounts in a secure way is no easy feat. Amazon provides facilities such as multi-factor authentication (MFA), password policies and cross-account credentials and role sharing, but setting all of those up correctly still is largely a task of combing through blog posts and best practice analysis. This document, with the help of the hardened concepts introduced at our client AutoScout24, is to illustrate how secure account setups on AWS are accomplished and what it takes to get from using single purpose, individual IAM accounts to a multi-cross-account role sharing concept. It will also tell you about steps to take to ensure compliance with company guidelines
AWS account are pretty easy to set up...sign up with your email, assign a password to your root account user, and you’re done. However, this root account user is an all-capable god within your AWS setup and should never be used except for certain provisional or billing tasks.
These are the core steps you must take in order to ensure the most basic level of security within your AWS account. Pretty much everyone should be running through these steps one by one, as they are a part of Amazon’s official documentation on how to secure IAM:
Sooner or later you will want to create multiple accounts to manage your resources on AWS, to improve overall security through separation of concerns. Having access to a user’s account credentials or tokens must not entail compromising all of your account’s resources or applications.
Once you have your two accounts talking to each other and Account 1’s users/groups are able to use Account 2 as their main driver for development try to think of further logical sharding within your account structure.
Accounts like “Production”, “Development”, “Staging” or “Playground” are a good idea. Compromising one of them doesn’t mean you have access to the other. Communication between the two should be avoided if possible, but can be enabled through VPC VPN endpoints or VPC peering. Use these features wisely and sparingly. The more you directly access resources across accounts the greater your attack surface becomes. For most purposes it should also not be necessary.
Ideally, you have your services also assume roles with temporary credentials across accounts and use the official, audited and hardened APIs supplied by Amazon. Tools like an AWS signing proxy make it even easier to ensure IAM is your one source of truth regarding credentials while retaining a sufficiently streamlined development process, even across scattered environments.
We would advise against delegating direct IAM write access to a continuous delivery role or integration environment, as those credentials could theoretically be obtained without supervision and used to take over IAM by a rogue actor or developer via manipulated artifacts or build instructions.
[Note: For the purposes of this listing password and MFA authentication in combination are considered secure and hard to compromise.]
To show you the advantages of the concepts we described earlier let’s take a look at a plausible (and usual) attack vector and how the steps we talked about earlier help to mitigate it:
All security measures are for nothing if you are not diligent about auditing relevant calls to protected systems and monitoring the impact of their actions. AWS provides you with a powerful tool capable of auditing most calls made to its API called CloudTrail. CloudTrail also allows for log events to be aggregated and stored in S3, which makes is possible to do dynamic processing based upon the collected events. By keeping a whitelist of roles, services and access policies you can then set up an automated way of discovering possible security incidents and be notified almost instantly.
The general design would look like this:
Be sure to enable CloudTrail for all regions, otherwise it will only log in the region you originally provisioned it in. With the number of regions AWS is available in steadily growing, creating/using unsupervised and probably undetected resources in an obscure region is a common vector of attack.
There are a number of ways to extend this system:
Netflix is famous for pioneering a lot of open source tools they build to run their infrastructure on AWS. Among them is Security Monkey. It’s a part of the Simian Army concept of automated auditing and compliance tools, with, currently, Chaos Monkey, Janitor Monkey and Conformity Monkey being the other three available.
Security Monkey by itself is still actively developed, albeit sparingly. It provides a service which is able to continuously scan your account(s) for possible violations of predefined policies and gradually either address these violations, acknowledge/justify them, or even classify whole accounts as “friendly” to alleviate the severity of the issues raised.
Security Monkey is also able to send you either daily or per-result email notifications, keeping you informed of possible security related changes in your account setup.
The best monitoring and auditing measures are useless without a continuous cycle of reviewing and resolving possible incidents. You need to establish a regular rotation of engineers dedicated to keeping track of the findings, actively engaged in resolving the incidents highlighted.
Better still, make it a part of a regular on-call schedule, so everyone has to assume responsibility from time to time.
Awareness is one thing, accurate, coordinated and timely response is another.
Managing secrets in distributed, heterogeneous environment can be very tasking. Luckily though, AWS provides you with a well integrated toolset for incorporating secret management into your workflow seamlessly and effortlessly. The service is called AWS KMS, or Key Management Service. The basics are pretty easy to explain: You can create a secret key, within AWS, which then lets you encrypt secrets. The result is a base64 encoded binary blob, which contains both information about the key needed to decrypt it, and the actual secret. You then need to explicitly allow users, groups and roles to access certain keys in KMS to encrypt/decrypt secrets.
This whole workflow allows you to use KMS to effectively manage secret credentials within AWS. KMS can be used to encrypt S3 buckets, RDS databases, CloudTrail logs and to use many more features of other services more securely.
Best practices should be separation of concerns, also when it comes to KMS keys, i.e. every team or even every application should get their own.
Not everyone in an organization is interested in or even shares the same beliefs concerning security. Some are outright negligent, some are zealots without regard for processes or business value. The key within an organization is to hit the right tone for everyone and find a fair compromise between “maximum security” and driving the business with security as an afterthought.
A successful model for addressing security concerns, especially with distributed, agile teams carrying individual responsible for separate business goals, has been to formalize the process of sharing security concerns, security related ideas and efforts as well as responding to strategic security requests in the form of an interest group.
A proposed solution would be the formation of a security interest group consisting of a limited number of individuals interested in security and security related topics, but at least one member from each team within the organization. These individuals form a team outside the confines of their current team’s structure and are meeting on a regular (mostly weekly) basis to discuss possible topics in and around a shared subject (in this case: security). They are also the proponent and watchdog when it comes to security within their team.
You can extend this role concept by also assigning each member of this group a superset of rights, delegating some security related tasks (e.g. creating roles and policies, creating keys or even users and groups) to its members, with a set of strict rules (e.g. a 4-eyes/6-eyes principle for roles/policies) attached to them.
Make sure members of the group are rotating continuously to avoid overexposure and fatigue. This also gives you the ability to spread knowledge more efficiently and effectively.
Interest group members have to hold themselves accountable against their own expectations. Furthermore, they will have to share vital aspects of their group discussions with their relevant teams on a regular basis.
Structuring the interest group around these key objectives, i.e. building an internally rooted community around security and security related topics, with advocates in every team, while delegating key tasks requiring an elevated level of security creates a foundation for an integrated thinking process which relates closely to a security minded but value-driven business.
It is essential that the organization gives itself and upholds a number of KPIs related to security. Examples could be “Outdated packages are updated within X days”, or “Security incidents are resolved within X hours”, or “A member of the security interest group acknowledges an open issue in X number of minutes”.
These KPIs are very important to combat the otherwise hard to come by negligence (after the broken window theory). However, rules and preventative measures are one thing. Making sure everybody sticks to them is another. Following up on KPIs in a regular and organized fashion (e.g. during stand-ups, incident reports and post-mortems) is just one of a few possible ways of implementing a culture with security as a first class citizen.
Should it come to the unfortunate event of a security related incident, conducting at least a post-mortem with scope of the people or the team involved and related to the security incident at hand should be a given. Getting everybody into the same room and discussing the outcome of the incident, what went wrong, what needs to be improved and preventative measures for the future.
Ideally, these post-mortems are made available to a much broader audience, be it on an organizational level, company wide or even involving your target audience or customers. The way the company GitLab is regularly doing public post-mortems on their blog and it has served them well in terms of community building, raising awareness and creating trust amongst their customer base. Sharing your failures (and a security related incident most certainly is a failure most of the time) and describing your action items, next steps and/or how you dealt with them is an opportunity to solidify your standing as a proponent of security and secure environment.
This strategy favors transparency over secrecy and increases awareness while improving education within the affected team, your organization and, if discussed publicly, with your customers and consumers.
Amazon's Web Services provide a lot of facilities to help securing environments, but it requires them to be understood and set up correctly. Identity and access management are difficult systems to get right, and AWS is no exception. Amazon’s solution reflects the inherent conflict of security and pragmatism since it’s neither straightforward nor easy to implement.
If you are unable to implement every recommendation in this document, try to get as far as you can and pick up the pace at a later point in time. Focus on the AWS recommended steps first and move on to the cross-account processes later.
Each and every one of them is a leap forward when it comes to the level of security you should strive for with your AWS cloud deployment.
Being aware of the shortcomings and tackling the most important issues when it comes to securing your AWS accounts is paramount and we hope this article will give a closer look at and understanding of what hardening your AWS account setup means and the incentive to drive your own process to secure your cloud infrastructure forward in a reasonable and effective way.
This is just the start.
Thoughtworkers Lisa Therese Junger, Ben Cornelius, Folker Bernitt, Cade Cairns, Daniel Somerfield and Johannes Müller (AutoScout24) have contributed to this article. Thank you!
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.