It’s easy to get started writing HashiCorp Terraform code to deploy and manage the infrastructure as code (IaC) for a single project, or even the infrastructure needed by a single engineering team. However, it gets a bit more complicated when the amount of Terraform code and automation grows to support an entire organizations infrastructure needs. When talking about governance, networking, application, and other infrastructure needs, it can get complicated very quickly to put all that Terraform code in a single monolithic project.

This article will walk you through several best practice strategies for organizing the HashiCorp Terraform infrastructure as code across an organizations teams and their source code repositories.

Organizing Terraform Project Code

The first decision to make when determining a HashiCorp Terraform organization strategy is to figure out how to actually manage the Terraform code that will be written. This is an important decision to make as it will determine how manageable and maintainable the Terraform code will be as the amount of code grows to support the organizations needs over time.

There are 3 primary strategies to organize HashiCorp Terraform code in multiple projects for easier maintainability:

• Break up Terraform code by Team Boundaries
• Break up Terraform code by Workload / Lifecycle
• Single Monolithic Terraform Project

Let’s take a look at what each of these organization strategy look like!

Break up code by Team Boundaries

The strategy of organizing Terraform code based on team boundaries can work great when each team is responsible for their own infrastructure deployments. This will organize all the infrastructure each team needs into their own individual Terraform project. Team boundary organization works great when it’s desired to have all the infrastructure for the team managed in a single code base.

Breaking up Terraform code by Team Boundaries provides individual teams with a central location to manage their infrastructure deployments.

The team boundary based Terraform projects do work well when individual teams are managing their own infrastructure. This is true so long as there are no cross-team boundaries broken where infrastructure gets shared between teams in some way. If this happens the strategy may become difficult to maintain, or just need to be broken in order to affectively manage the shared infrastructure.

Break up code by Workload / Lifecycle

The strategy of organizing Terraform code based on workload or lifecycle boundaries is amore granular approach than the team boundaries strategy. Decoupling the Terraform code in this way means that each application / workload, or just infrastructure resources that are provisioned and eventually deleted together will get managed together. This provides for a more organized approach to managing the Terraform code in a way that changes to the infrastructure and deployments for a single workload will be isolated from having unintended / accidental effects on any other.

Breaking up Terraform code by Workload / Lifecycle can provide for a more organized strategy made up of smaller isolated Terraform projects.

If the team or organization wants to have a more structured strategy for Terraform code organizations, then this isolation by workload / lifecycle will work really well. This will also keep Terraform projects as small as possible, which can help reduce the risk of user error when editing the Terraform code and help increase reliability that any infrastructure changes wont have any adverse side effects. Keep in mind that this strategy may also increase the amount of overhead in setting up automation or even in longer term maintenance of the teams Terraform code.

Break up by Workload

Breaking up Terraform code into multiple projects where each workload has it’s own project is a great way of organizing the code based on the each application or system. This will mean each project contains all the Terraform code to declare the infrastructure for a single application workload and nothing further.

The following are some example breakouts of Terraform projects based on individual workloads:

• Application 1
• Application 2
• Database Server 1
• Data Warehouse
• Data Processing Pipeline

When deciding the workload boundaries you’ll need to ensure there are no cross dependencies between workloads. For example, this doesn’t work well if a ‘Storage Account’ is shared between ‘Application1’ and ‘Application2’, when both apps need to access that same ‘Storage Account’. If the Storage Account resource is shared, then it may warrant it’s own Terraform project for just that Storage Account, or even merging ‘Application1’ and ‘Application2’ into the same project.

Break up by Lifecycle

Breaking up Terraform code into multiple projects based on the management lifecycle of the resources is a great way to organize Terraform code for larger and more complex deployments. This is a strategy that works for larger organizations and enterprises that manage a large amount and variety of infrastructure.

Management Lifecycle generally means infrastructure resources that are managed together. This can certainly mean breaking code up by workload, but it can also mean breaking code up where a workload may have multiple Terraform projects for different sets of infrastructure of the larger workload as a whole. It could also mean breaking up Terraform infrastructure code just differently based on which resources are deployment, managed, and configured together.

When organizing infrastructure resources that are deployed, managed, and configured together, if may mean breaking that code out into different Terraform projects by type of cloud resource, or even based on the security or management plane access controls.

An example of breaking up Terraform infrastructure code into separate projects based on management lifecycle would be breaking out all the networking components into a single project. Networking components often have dependencies of how they connect to each other, and the organization of Subnets and IP address ranges (CIDR ranges). These may make sense to group together based on management lifecycle.

Databases are another example that may make sense to break up based on management lifecycle. The virtual machine or other compute infrastructure is generally secured and managed separately from database infrastructure. This is especially true when it’s necessary to secure data access, management, and backup. In this case it may make sense to manage the Terraform code for database infrastructure in a separate project.

Larger Terraform infrastructure deployments generally benefit best from organizing the code by management lifecycle. This means breaking up the code on some boundaries of infrastructure managed together based on relationship / dependencies, security, specific workloads, or even multiple projects per workload.

The following is an example of how a more complex infrastructure deployment may be broken up in to multiple Terraform projects based on management lifecycle:

• Governance
  A governance project would be a Terraform project that contains all the infrastructure code that sets up the organization and compliance rules of cloud infrastructure. For example, this would be things like Azure Resource Groups for grouping deployed resources, or Azure Policies that enforce organizational standards in the cloud.
• Networking
  A networking project would contain all the Terraform code for deploying and managing the networking infrastructure for the organization.
• Single Application
  A single Terraform project for a single application workload would make sense to deploy and manage all the infrastructure resources for a particular application; so long as that applications resources are all managed together.
• Application Compute
  The compute resources for a single application workload may make sense to have their own Terraform project for deployment if they are managed separately from other infrastructure types; like databases or storage.
• Application Database and Storage
  The database and storage for an application workload may make sense to have their own Terraform project for deployment if they are managed separately from other infrastructure types; like compute.
• Data Warehouse
  Often times a data warehouse is a stand-alone Terraform project as it contains resources that are specific to the data warehouse workload, and the data warehouse may be connected to and integrated with other workloads deployed with multiple different Terraform projects. Data warehouses generally have their own security and management lifecycle that’s very different from other applications and infrastructure, so data warehouse infrastructure generally makes sense to have dedicated Terraform project just for this.

Single Monolithic Terraform Project

Organizing the HashiCorp Terraform code for the team or organization into a single project can be done. This is the simplest strategy for managing the Terraform project code. It works well if the Terraform code is centrally managed by a single team of SREs or DevOps Engineers that are responsible for the infrastructure deployment and configuration.

A Single Monolithic Terraform Project is the easiest way to get started managing infrastructure code, and generally works well when a single engineering team is responsible for infrastructure deployments.

As the Terraform code base increases and the complexity of the Terraform project grows, this is the least maintainable strategy for organizing all the Terraform IaC code of the team. Even though this may not be the strategy for large code bases, it’s still a perfectly valid strategy when appropriate.

Wrap Up

Multiple projects may mean multiple source control repositories. Either way, multiple Terraform projects will mean the need for multiple CI/CD deployment pipelines. The tips in this article should help in determining a HashiCorp Terraform code organizational strategy that lends itself to better maintainability over time. There really is not any single best Terraform code organizational strategy that is best. The chosen strategy for a team to standardize on will need to fit the teams unique requirements. It also influences the overall direction of the DevOps workflow used for managing the teams infrastructure deployments. Ultimately, the organization strategy that is chosen may be a blended combination of all these strategies, so long as it meets the needs of the team or enterprise for long-term infrastructure management.

About the Author