integrate jenkins on terraform with bitbucket

Question

I have created terraform script for aws architecture which includes ec2 instance and jenkins on ec2. I am new to jenkins and trying to figure how to integrate it with bitbucket using the existing terraform script. Any help would be greatly appreciated.

Answer 1

CI/CD pipeline workflow for applying changes to the infrastructure using terraform:

1. Developer or Operations Engineer change the terraform configuration file in his local machine and commit the code to BitBucket.
2. Gitbucket webhook triggers a continuous integration job to jenkins.
3. Jenkins will pull the latest code from the configured repo which contains terraform files to its workspace.
4. It reads the terraform configuration then initialize the remote consul backend.
5. Terraform generates a plan about the changes that has to be applied on the infrastructure
6. Jenkins send notification to a slack channel about the changes for manual approval.
7. Here, the user can approve or disapprove the terraform plan.
8. The user input is sent to jenkins server for proceeding with the further action.
9. Once the changes are approved by an operator, jenkins will execute terraform apply command to reflect the changes to the infrastructure.
10. Terraform will create an report about the resources and its dependency created while executing the plan.
11. Terraform will provision the resources in the provider environment.
12. Jenkins will again send a notification to the slack channel about the status of the infrastructure after the applying changes on it. Once the job is executed, Jenkin pipeline job is configured to clean up the workspace created by the job.

How to setup the deployment environment?

1. Create a repo in scm tools like gitlab or bitbucket and commit the terraform configuration and its dependency module to the repo. If you are using any third party remote module as a dependency, it will be automatically downloaded while execution.
2. If you do not have Jenkins server, then just pull a jenkins docker image and run it in your local machine. If you are setting it up in cloud environment, check the jenkins virtual machine image from marketplace to set up the environment and configure the required plugins.
3. Create a webhook in your bitbucket repo settings to invoke a http call to your jenkins call back url for triggering continuous integration job.
4. If you have an existing jenkins server, ensure pipeline plugin is installed in the jenkins server. Otherwise goto "Manage plugins" and install pipeline plugin.
5. In this project, we are using consul as a remote backend for state storing and state locking. It is not recommended to use local state for the case where multiple people involved in the project and for production deployments. It is good to use remote backend which provides highly available storage with state lock functionalities to avoid writing the state by multiple users at a time.
6. If you do not have consul key-value store in your environment, just pull consul docker image and setup a single node cluster. If it is production deployment, setup a distributed key-value store.
7. Create an application in slack and note down the slack integration details for configuring it in Jenkinsfile.
8. Configure your provider details and backend details in main terraform configuration file either by environment variable or persisting in a repo. In my case, I am going to provision a resource in AWS and my CI server is hosted in AWS. So I am assigning an IAM role to my server with sufficient privileges.
9. Create a new project in Jenkins by using pipeline plugin.
10. Add the Jenkinsfile where the pipeline stages are defined. Save the job and trigger it manually for testing. Then apply changes to the configuration and commit the changes to the bitbucket and ensure the job is automatically triggered. Check Jenkins log for more details about the job.

###Jenkinsfile###
import groovy.json.JsonOutput

//git env vars
env.git_url = 'https://user@bitbucket.org/user/terraform-ci.git'
env.git_branch = 'master'
env.credentials_id = '1'

//slack env vars
env.slack_url = 'https://hooks.slack.com/services/SDKJSDKS/SDSDJSDK/SDKJSDKDS23434SDSDLCMLC'
env.notification_channel = 'my-slack-channel'

//jenkins env vars
env.jenkins_server_url = 'https://52.79.46.98'
env.jenkins_node_custom_workspace_path = "/opt/bitnami/apps/jenkins/jenkins_home/${JOB_NAME}/workspace"
env.jenkins_node_label = 'master'
env.terraform_version = '0.11.10'

def notifySlack(text, channel, attachments) {
    def payload = JsonOutput.toJson([text: text,
        channel: channel,
        username: "Jenkins",
        attachments: attachments
    ])
    sh "export PATH=/opt/bitnami/common/bin:$PATH && curl -X POST --data-urlencode \'payload=${payload}\' ${slack_url}"
}

pipeline {
	agent {
		node {
			customWorkspace "$jenkins_node_custom_workspace_path"
			label "$jenkins_node_label"
		} 
	}
	
	stages {
		stage('fetch_latest_code') {
			steps {
				git branch: "$git_branch" ,
				credentialsId: "$credentials_id" ,
				url: "$git_url"
			}
		}

		stage('install_deps') {
			steps {
				sh "sudo apt install wget zip python-pip -y"
				sh "cd /tmp"
				sh "curl -o terraform.zip https://releases.hashicorp.com/terraform/'$terraform_version'/terraform_'$terraform_version'_linux_amd64.zip"
				sh "unzip terraform.zip"
				sh "sudo mv terraform /usr/bin"
				sh "rm -rf terraform.zip"
			}
		}

		stage('init_and_plan') {
			steps {
				sh "sudo terraform init $jenkins_node_custom_workspace_path/workspace"
				sh "sudo terraform plan $jenkins_node_custom_workspace_path/workspace"
				notifySlack("Build completed! Build logs from jenkins server $jenkins_server_url/jenkins/job/$JOB_NAME/$BUILD_NUMBER/console", notification_channel, [])
			}
		}

		stage('approve') {
			steps {
			  notifySlack("Do you approve deployment? $jenkins_server_url/jenkins/job/$JOB_NAME", notification_channel, [])
				input 'Do you approve deployment?'
			}
		}

		stage('apply_changes') {
			steps {
				sh "echo 'yes' | sudo terraform apply $jenkins_node_custom_workspace_path/workspace"
				notifySlack("Deployment logs from jenkins server $jenkins_server_url/jenkins/job/$JOB_NAME/$BUILD_NUMBER/console", notification_channel, [])
			}
		}
	}
	
	post { 
  	always { 
    	cleanWs()
   }
  }
}
###Code Completed###