– EasyYouCloud –

Before creating your client, you need to create a PAT ( Personal access Token), so that your application can call azure devops and interact with it in a REST manner.

But before going further, a good practice is to use a service account created in Azure Active Directory with the role DevOps Administrator :

Now that you have the role and logged with it, you can go straight to Azure DevOps.. :

Give it a name and allow some right to it. In my case, I want to start some release pipeline when I am going to trigger calls..

Make sure you save the token…

Let’s code now. Create the client using your favorite language ( python SDK source here ) . first you install the library :

pip install azure-devops

Then code client will be :

from azure.devops.connection import Connection from msrest.authentication import BasicAuthentication import pprint 

# Fill in with your personal access token and org URL 

personal_access_token = 'YOURPAT' organization_url = 'https://dev.azure.com/YOURORG' 

# Create a connection to the org 

credentials = BasicAuthentication('', personal_access_token) connection = Connection(base_url=organization_url, creds=credentials) 

# Get a client (the "core" client provides access to projects, teams, etc) 

core_client = connection.clients.get_core_client() # Get the first page of projects get_projects_response = core_client.get_projects() index = 0 while get_projects_response is not None:     for project in get_projects_response.value:         pprint.pprint("[" + str(index) + "] " + project.name)         index += 1     if get_projects_response.continuation_token is not None and get_projects_response.continuation_token != "":         

# Get the next page of projects

get_projects_response = core_client.get_projects(continuation_token=get_projects_response.continuation_token)     else:         # All projects have been retrieved         get_projects_response = None

And… voila :

Now the fun part, let’s trigger a pipeline !

Here is the synthax to run a pipeline :

  
 
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
import pprint



# Fill in with your personal access token and org URL
personal_access_token = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX'
organization_url = 'https://dev.azure.com/mouradcloud'
project_name = "IaCSpokeStorageNFS1"
pipelines_name ='IaCSpokeStorageNFS1-CI-preprod'
pipelines_id =13



# Create a connection to the org
credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)



# create a Get a pipeline client 
def create_pipeline_client():
    # Get connection from devOps & create pipeline client
    pipeline_client = connection.clients_v5_1.get_pipelines_client()
    return pipeline_client



# Get the first page of pipeline
pipeline_client = create_pipeline_client()
pipeline_list_response = pipeline_client.list_pipelines(project_name)



def run_pipe(project, pipe_id):
    #run_params = {
    #    'variables': '{"var_test": "abc"}'
    #}
    run_params = {}
    pipeline_client = connection.clients_v6_0.get_pipelines_client()
    pipeline_client.run_pipeline(project=project, pipeline_id=pipe_id, run_parameters=run_params)



for pipeline in pipeline_list_response.value:
    print "I am going to run  pipeline : " 
    pipeline_name = pipeline.name
    print pipeline_name
    pipeline_url = pipeline.url
    print pipeline_url
    pipeline_id = pipeline.id
    print pipeline_id
    print "**** debug starting run ****"
    run_pipe(project_name, pipelines_id)
    print "**** debug ending run ****"


 

Go on AAD portal and select New guest users :

Invite user and add some more details along, like a welcome message. Make sure to tell them to use same password as onprem.. ( just in case )

User receive the mail with message on the current solution and onprem tenant :

User go to the link and validate ( user will create put the password ) :

User receive a 2 way validation with a secret number :

User got notified that he has an ID in AAD now :

He can check on your AAD his accreditation :

And lastly log into any azure related app using his identity on your tenant :

As you see a default tenant is created if no one exist in azure with same :

Still, he belongs to the 2, he just need to switch when in user portal :

Here is the example of a switch on the 2

In this post we are going to discuss on how we could leverage azure devops to do IaC as code. there is plenty ways to address the use case, but I will do easiest way here. first step will be to use it for CI. For CD, Ansible will be used along with Azure devops.

End target is to create and operates HUBs ( prod and pre-prod in a CI-CD fashion )

In this post we are going to discuss on how we could leverage azure devops to do IaC as code. there is plenty ways to address the use case, but I will do easiest way here. first step will be to use it for CI. For CD, Ansible will be used along with Azure devops.

Start creating an Azure Devops projects. Since we are going to simulate production and pre-production environment for IaC, we are going do some segregation for compliance and governance concerns.

Assuming you have created an organization already, I created the project IaCHUBv1 :

I went on github and created 2 private repos (empty for the moment ) , one for production and second for pre-production, As I want to have the possibility to manipulate each one at will.

• 

• 

No I go to Azure and I am going to use 2 subscriptions, one for production and other one for pre-production environment

On those 2 subscriptions, I am going to create 2 services principals that are going to be the identity that will do deployment. You can use this command line to check :

az account list --output table

This will give something like this :

I am going to select to first and third with this ( you will need to repeat the to command below 2 times, one for each subscriptions ) :

az account set --subscription <Azure-SubscriptionId>
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/SUBSCRIPTION_ID" 

This will give you the service principals ( example for the first one ) :

These values will be mapped to the Terraform variables this way, note those value in a safe place for the moment ( you will not be able to get them back ) :

• appId (Azure) → client_id (Terraform).
• password (Azure) → client_secret (Terraform).
• tenant (Azure) → tenant_id (Terraform).

Now go to the terraform plugin for azuredevops and just install it to you organisation

Or the official one works too :

Install Git on your laptop (using linux kernel on Windows 10 is fine ) :

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install git

We need to authenticate to Azure DevOps using SSH keys. If we have SSH keys, we can skip this step, and jump to the next one.

ssh-keygen -C "email@domain.com"

The process will generate two files id_rsa and id_rsa.pub files. The first one is the private key (don’t share it) and the second one is the public key.

After we generated the SSH keys, it is time to upload them to Azure DevOps.
We open the Azure DevOps website, click on our profile picture, then click on 3 dots and finally click on user settings option.

On the menu, we click on the SSH public keys option:

Now we add the empty repos previously created :

• 

• 

Repeat the operation for every repo you want to add

Thenk go on the files section to get url of files :

No I go to my laptop were my code is sitting and I will git init the location

I will type those from the location I selected

git init
git remote add origin git@ssh.dev.azure.com:v3/mouradcloud/IaCHUBv1/iachubv1preprod
git add .
git commit -m "commit with terraform files upload initial"
#there is a readme file in the repo, you first need to pull before pushing :
git pull git@ssh.dev.azure.com:v3/mouradcloud/IaCHUBv1/iachubv1preprod
#then push
git push --set-upstream origin master

You will be able to see uploaded files :

Creating now the Azure pipeline to deploy our IaC

Then select source :

then empty jobs :

We will change the name of the Agent. As for now, we will be using Azure devops free build agent. We will be able to add agent pool later if more power is needed, but this is not free, you will need to declare VMs that are going to do it.

Then we will add a task to copy files :

We choose the Repos source folder, and select to copy all content. We will set the target folder as : $(build.artifactstagingdirectory)/Terraform

We click the plus sign (+) to add a new job. Type Publish Build Artifacts and leave it with default parameters:

There is the possibility to use the following option that is going to trigger a CI when the source repo is modified and changes are committed ( we will not activate this for the HUB, there is no special interest since this is a near-static environment. However, we can Create new CIs using sub folder in the terraform modules if we want to add minor changes into the configuration.

Then Queue it for the CI to be collected by an available agent than is going to be run

It is creating a build for you, which is basically, nothing more than copy files from the repo to the agent that will run later tasks.

Now, while checking the logs :

You should receive an email telling you that your build is successful.
If the status of the job is Sucess, we are ready for the next step, where we are going to create the Release Pipeline.

On this stage, we will use the artifact generate on the build pipeline and create a Stage task with these following tasks:

• Terraform Installer (Install Terraform)
• Terraform CLI (Terraform Init)
• Terraform CLI (Terraform Validate)
• Terraform CLI (Terraform Plan)
• Terraform CLI (Terraform Apply)

For all these tasks we will use the Terraform Build & Release Tasks extension from the Marketplace (installed on previously).

In the Select a template page, we choose an Empty job template:

Then we click on the Add an artifact button.

In the Add an artifact page, we choose the Build button and configure the Source (build pipeline) to use the build pipeline created on the previous step.

We click the Add button, DO NOT (yet) click on the lightning icon and activate the CD (Continuous Deployment), we keep this of other use not for HUBs.

We close the Continuous deployment trigger page and rename the pipeline:

Now, we need to configure the Stages. Click on the Stage 1 button to rename the stage name.

We close the Stage name page and then click on the 1 job, 0 task link on Terraform deployment button.

We click on the plus sign (+), next to the Agent job and search for terraform.

We select the Terraform Installer task and click on the Add button next to it.

The Terraform Installer task is added with the latest version of Terraform, but I am changing it to another :

Then we add terraform command for init . The terraform init command is used to initialize a working directory containing Terraform configuration files. This is the first command that should be run.

We select the Terraform CLI task and click on the Add button next to it, in this step, we will configure Terraform CLI for Terraform Init.

Configure the init Command, the Configuration Directory to use the drop/Terraform folder of the Build Pipeline and select azurerm in the Backend Type dropdown.

We have not configure the AzureRM yet, so this is the perfect moment.

Expand the AzureRM Backend Configuration and select an existing Azure Subscription. If we don’t have an Azure Subscription configured, we click on + New button to configure one.

It is taking you to the connections pages :

Then, we select the Service principal (manual) option.

Then you will enter the above information that you received when you created the service principals :

After registering all connections, you will see them available there :

Then, we configure the Azure Remote Backend and we have a few options:

• Use the Create Backend (If not exists) option
• Use an existing Remote Backend created with scripts if one existes
• Automate the process adding an extra task on the pipeline.

In order to validate, we will just start the release to make sure every thing is working fine, we will do the terraform apply afterwards

So it created a Storage account in a ressource group for service usage and this will be used to store the state of terraform files config ( not that it is windows logs… I forgot to validate to linux ubuntu for agent… I will correct this ) .

Let’s go now to the terraform validate, plan and apply

However, I will do it in a 2 steps :

• Step 1 : Terraform Init validate and plan
• Step 2 : adding terraform apply in a second task and I want to be sure to have no error before applying changes ( this is useful if i want to bring management validations )

Also, (optional but recommanded) you can adjust the retention, otherwise you will find yourself with a lot of artifacts , releases etc and you might get lost rapidly..

Optional Too : you can report to Azure Devops boards the release pipeline tasks :

Unfortunately, for the second task we need to bring a second “init” step because we need to make the terraform job to have the details for state :

Do not foreget to set the second task to manual otherwise the deployment will be automatic :

then create the release :

Seems OK ……

Till the end of the deployment :

Just to finish this, we can use keyvault to store secrets and call the secrets using environment variables. If secrets are present, you just need to refresh and you will see it.

Remember we created an azure key vault thru Terraform and updated a secret thru it.

For the sake of the demonstration, I will add a new secret thru Az shell portal.

first we need to get user shell data :

az ad user show --id <email-address-of-user> 

You will be looking for the “objectId” value, or just :

az ad user show --id <email-address-of-user> --query "objectId  

Then allow your user to manipulate :

az keyvault set-policy -n <your-unique-keyvault-name> --spn <ApplicationID-of-your-service-principal> --secret-permissions get list set delete --key-permissions create decrypt delete encrypt get list unwrapKey wrapKey

Then finally create a secret entry :

az keyvault secret set --vault-name "ccppdhbkvlt" --name "badpassword" --value "hVFkk965BuUv"

Then you select the secret you want to import :

Then save and go to your release pipeline the variable section and link rhe variable group that is in keyvault :

Now you just need to use the variable ‘badpassword’ in the variable.tf and variable.tfvars files and you are good to go. This can be typically used to store password for VMs for instance..