Setup the virtual machine Linux and Windows for Agents and Azure-Pipelines

tags: research devops azure

On the mood for sharing, This is second blog i am sharing for today, come and read about Setup MySQL with Wordpress in k8s: Easy migrate or not !!. So if you want to know about how to setup the Linux and Window VM for Azure-Pipeline, Go check it out downbelow

What things are you looking for ?

• Azure Pipelines is one most of things Azure, I confess that pretty supercool than other thing alternative like Gitlab CI/CD, Github Action
• Why it good ? Template by what you want is why i like its, sofar it will help you optimize the time for customize new pipeline script but not make effect for another.
• But for doing this things and more secure, more time, more customize, Azure users will prefer how to use selfhost instead of vm provided by azure for purpose running the pipeline CI/CD on this Agents.
  

The scenerio for setup the agent and more things about it

• VM is Linux or Windows can be have access by azure pipeline via token, It will contains the environment for doing with Azure Resource like create VM, interaction with K8s - Cluster so you need about role for it or Azure AD for creating Application Role for access anything via that IAM.
  
• VM agents is environment which run anything task on pipeline so you need to configure them to have exacty what this pipeline need so go details to knowing about it.
• How to connect anything VM, Agent or VM cluster in Azure, that things will optimize your cost to setting up bastion host for service. So i will share about this on this Blog. So let go and break down anything from small task in down below.

Step to setup

Prepare for setup the azure agent

• PAT (Personal Access Token) - most of important thing, which you need to configuration you VM

  1. You need to create Azure DevOps that one will intergration Azure Pipeline inside

  2. Go for security tab or setting tab on left your avatar icon to creating for your own Token

  3. Token will include what role for access via itself, So i will drop down this

     Rule	Value	Rule	Value	Rule	Value
     Agent-Pools	Read & Manage	Load Test	Read	User Profile	Read
     Build	Read & Execute	Marketplace	Acquire & Manage	Variable Group	Read
     Code	Read & Status	Notification	Read	Task Groups	Read
     Connected server	Connected Server	Packaging	Read & Write	Team Dashboard	Read
     Deployment Groups	Read & Manage	Project and Team	Read	Test Management	Read
     Environment	Read & Manage	Release	Read,write & execute	Wiki	Read
     Extension Data	Read	Secure Files	Read, create & manage	Work items	Read
     Extensions	Read	Security	Manage
     Identity	Read	Service Connection	Read & Query

  4. After you have applied rule you will have the PAT token and choose the expire day which you want (1 year is limited)

• Go to the settings page of TFS on symbol in the edge and Choose create a new agent pool or existed pool

• After create the pool you want have to create new agent, the agent configure have 3 OS can provide is Win, Mac and Linux. So you can choice once and do with the manual

NOTICE: You need to take care about this situation for prevent PAT can access to your pool

1. If you don't have adminastrator of agent-pool, when you created a new agent in pool with your token –> It will cause error so remembering you need to have role adminastrator of pool (Not inherited role, yoi need adminastrator for whole pool from Org into repo, if you don't have it your progress for provisioning will fail)

Solution: Add role adminastrator for who own token created and give that one for progress which create the agent

2. Validate the token when expire or delete

Solution: Do regenerate token again

Prepare job is done, you complete 1/2 way to having the goal. Go create IaC and provisioning your VM (Linux or Windows)

Linux

• With linux OS - Ubuntu 22.04 for example, it quite easily so for optimize time you don't need to go far with bulild own image, Just do with raw image which Azure Provide
  

• You can running that kind on your own VM or create that to Docker, Which my situation for customize more than one runner in VM, Docker is prefer more than first optional
  

• I will choose terraform and like i said go for check my Terraform session to know about more. I just put my code about main.tf and one more, you need to know about to create recycle your code in mutiple environment with Terraform, Go this to understand the theory.

• So i have agents-pools folder to put that customize for my VM agents provide and second one modules, this kind will put basically what resource i want to provision for each of vm

• With modules, i will have 3 things inside network, bastionhost, vm

  
  So go for details on each modules. One more things i will put whole template for Window and Linux, so you need to consider to find what you want

network: This kind contain what network need to configure for vm subnet, ip, rule security of firewall, ...

modules/network/main.tf

resource "azurerm_virtual_network" "base" {
  name                = var.pool_name
  address_space       = var.address_space
  location            = var.location
  resource_group_name = var.resource_group_name
  tags                = var.tag
}

resource "azurerm_subnet" "base" {
  name                 = "${var.pool_name}Subnet"
  resource_group_name  = var.resource_group_name
  virtual_network_name = azurerm_virtual_network.base.name
  address_prefixes     = var.address_prefixes_agent_subnet
  depends_on = [ azurerm_virtual_network.base ]
}

resource "azurerm_subnet" "bastion_host" {
  count                = var.bastion_host_checked == true ? 1 : 0
  name                 = "AzureBastionSubnet" # Can not change this name
  resource_group_name  = var.resource_group_name
  virtual_network_name = azurerm_virtual_network.base.name
  address_prefixes     = var.address_prefixes_bastion_subnet
}

resource "azurerm_network_security_group" "base" {
  name                = var.pool_name
  location            = var.location
  resource_group_name = var.resource_group_name
}

resource "azurerm_network_security_rule" "ssh" {
  count                       = var.type_os_vm == "linux" ? 1 : 0
  name                        = "${var.pool_name}SSHrule"
  priority                    = "100"
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "22"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = var.resource_group_name
  network_security_group_name = azurerm_network_security_group.base.name
}

resource "azurerm_network_security_rule" "rdp" {
  count                       = var.type_os_vm == "windows" ? 1 : 0
  name                        = "${var.pool_name}RDPRule"
  priority                    = "100"
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "3389"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = var.resource_group_name
  network_security_group_name = azurerm_network_security_group.base.name
}

resource "azurerm_network_security_rule" "winrm" {
  count                       = var.type_os_vm == "windows" ? 1 : 0
  name                        = "${var.pool_name}WinRMRule"
  priority                    = "110"
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "5985"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = var.resource_group_name
  network_security_group_name = azurerm_network_security_group.base.name
}

resource "azurerm_subnet_network_security_group_association" "base" {
  subnet_id                 = azurerm_subnet.base.id
  network_security_group_id = azurerm_network_security_group.base.id
}

resource "azurerm_public_ip" "base" {
  name                = var.pool_name
  resource_group_name = var.resource_group_name
  location            = var.location
  allocation_method   = "Static"
  sku                 = "Standard"
}

resource "azurerm_network_interface" "base" {
  count               = var.checked_scaleset == false ? 1 : 0
  name                = var.pool_name
  location            = var.location
  resource_group_name = var.resource_group_name

  ip_configuration {
    name                          = var.pool_name
    public_ip_address_id          = var.bastion_host_checked == true ? null : azurerm_public_ip.base.id
    subnet_id                     = azurerm_subnet.base.id
    private_ip_address_allocation = var.private_ip_address_allocation
  }
}

On my source code, i have additional condition for create bastion host if you want. And that will help you specify via variables. You need to know about terraform and it will break down this concept. "This is file will overwrite the variables.tf file and need specify parituclar variables need to be updated"

Example:

terraform.tfvars
    ## Value need to specify included:
    azure_pat = "Token123"
    pool_name = "AutomationTest"
    url_org = "https://dev.azure.com/testorg"

    ## But it will have overwritten variables, you can specify by 
    ## read that in variables.tf on each module and inside folder create new agent

bastion_host: That kind like stuff supersecure for connect VM with middle VM managed by Azure. This will have cost to much for operating them (Consider about that)

modules/bastion_hosts/main.tf

resource "azurerm_bastion_host" "basepool" {
  name                = var.pool_name
  location            = var.location
  resource_group_name = var.resource_group_name

  ip_configuration {
    name                 = "Configuration"
    subnet_id            = var.subnet_id
    public_ip_address_id = var.public_ip_address_id
  }
}

Quite easily

, that just create bastion host the network is taken care by network above. So go for check it on important part VM or Virtual Machine

vm: That is your machine for operating your code, your pipeline and your extender task

modules/vm/main.tf

# Create a credential for linux machines
resource "tls_private_key" "base" {
  count     = var.type_os_vm == "linux" ? 1 : 0
  algorithm = "RSA"
  rsa_bits  = "2048"
}

# Create a credential for windows machines
resource "random_password" "base" {
  count            = var.type_os_vm == "windows" ? 1 : 0
  length           = var.length_of_sensitive
  special          = var.checked_special_characters
  override_special = var.override_special_characters
}

# Create the Linux virtual machine
resource "azurerm_linux_virtual_machine" "base" {
  count                 = ((var.type_os_vm == "linux") && (var.checked_scaleset == false)) ? 1 : 0
  name                  = var.pool_name
  computer_name         = var.vmname
  resource_group_name   = var.resource_group_name
  location              = var.location
  size                  = var.agent_vm_size
  admin_username        = var.admin_username
  network_interface_ids = [var.nic_id]
  os_disk {
    name                 = var.pool_name
    caching              = var.agent_vm_caching
    storage_account_type = var.agent_vm_storage_account_type
  }

  admin_ssh_key {
    username   = var.admin_name_for_ssh
    public_key = tls_private_key.base[0].public_key_openssh
  }

  source_image_id = var.image_id

  dynamic "source_image_reference" {
    for_each = var.image_id == null ? tolist([1]) : []
    content {
      publisher = var.configuration_vm.publisher
      offer     = var.configuration_vm.offer
      sku       = var.configuration_vm.sku
      version   = var.configuration_vm.version
    }
  }

  disable_password_authentication = true

  identity {
    type = "SystemAssigned"
  }

  user_data = base64encode(
    templatefile(
      "${dirname(dirname(abspath(path.module)))}/agent-pools/${var.pool_folder}/data/userdata.tpl",
      {
        version  = var.agent_version,
        url      = var.url_org,
        auth     = var.auth_type,
        token    = var.azure_pat,
        pool     = var.pool_name,
        agent    = var.agent_name,
        workdir  = var.workdir,
        tagagent = var.tagagent
      }
    )
  )
  tags = var.tag
}

resource "azurerm_linux_virtual_machine_scale_set" "base" {
  count               = ((var.type_os_vm == "linux") && (var.checked_scaleset == true)) ? 1 : 0
  name                = var.pool_name
  resource_group_name = var.resource_group_name
  location            = var.location
  sku                 = var.agent_vm_size
  instances           = var.vmss_instances
  admin_username      = var.admin_username
  tags                = var.tag

  user_data = base64encode(
    templatefile(
      "${dirname(dirname(abspath(path.module)))}/agent-pools/${var.pool_folder}/data/userdata.tpl", {
        version  = var.agent_version,
        url      = var.url_org,
        auth     = var.auth_type,
        token    = var.azure_pat,
        pool     = var.pool_name,
        agent    = var.agent_name,
        workdir  = var.workdir,
        tagagent = var.tagagent
      }
    )
  )

  identity {
    type = "SystemAssigned"
  }

  admin_ssh_key {
    username   = var.admin_name_for_ssh
    public_key = tls_private_key.base[0].public_key_openssh
  }


  os_disk {
    storage_account_type = var.agent_vm_storage_account_type
    caching              = var.agent_vm_caching
  }

  network_interface {
    name    = var.pool_name
    primary = true

    ip_configuration {
      name      = "internal"
      primary   = true
      subnet_id = var.subnet_id
    }
  }

  source_image_id = var.image_id

  dynamic "source_image_reference" {
    for_each = var.image_id == null ? tolist([1]) : []
    content {
      publisher = var.configuration_vm.publisher
      offer     = var.configuration_vm.offer
      sku       = var.configuration_vm.sku
      version   = var.configuration_vm.version
    }
  }
}

# Create a window virutal machine
resource "azurerm_windows_virtual_machine" "base" {
  count               = ((var.type_os_vm == "windows") && (var.checked_scaleset == false)) ? 1 : 0
  name                = var.pool_name
  resource_group_name = var.resource_group_name
  location            = var.location
  size                = var.agent_vm_size
  computer_name       = var.vmname
  admin_username      = var.admin_username
  admin_password      = random_password.base[0].result
  network_interface_ids = [
    var.nic_id
  ]
  os_disk {
    caching              = var.agent_vm_caching
    storage_account_type = var.agent_vm_storage_account_type
  }

  identity {
    type = "SystemAssigned"
  }

  source_image_id = var.image_id

  dynamic "source_image_reference" {
    for_each = var.image_id == null ? tolist([1]) : []
    content {
      publisher = var.configuration_vm.publisher
      offer     = var.configuration_vm.offer
      sku       = var.configuration_vm.sku
      version   = var.configuration_vm.version
    }
  }
}

resource "azurerm_windows_virtual_machine_scale_set" "base" {
  count               = ((var.type_os_vm == "windows") && (var.checked_scaleset == true)) ? 1 : 0
  name                = var.pool_name
  resource_group_name = var.resource_group_name
  location            = var.location
  sku                 = var.agent_vm_size
  instances           = var.vmss_instances
  admin_password      = random_password.base[0].result
  admin_username      = var.admin_username

  os_disk {
    storage_account_type = var.agent_vm_storage_account_type
    caching              = var.agent_vm_caching
  }

  network_interface {
    name    = var.pool_name
    primary = true

    ip_configuration {
      name      = "internal"
      primary   = true
      subnet_id = var.subnet_id
    }
  }

  source_image_id = var.image_id

  dynamic "source_image_reference" {
    for_each = var.image_id == null ? tolist([1]) : []
    content {
      publisher = var.configuration_vm.publisher
      offer     = var.configuration_vm.offer
      sku       = var.configuration_vm.sku
      version   = var.configuration_vm.version
    }
  }
}

# Virtual Machine Extensions (Configurations only for single instance - Do not test for scaleset)
# Only for Linux Machine
resource "azurerm_virtual_machine_extension" "bootstrap_script_file" {
  count                = ((var.checked_extension == true) && (var.type_of_extension_bootstrap == "template")) ? 1 : 0
  name                 = var.name_extension
  virtual_machine_id   = azurerm_linux_virtual_machine.base[0].id
  publisher            = "Microsoft.Azure.Extensions"
  type                 = "CustomScript"
  type_handler_version = "2.0"
  settings = jsonencode({
    "script" : "${base64encode("${file("${dirname(dirname(abspath(path.module)))}/agent-pools/${var.pool_folder}/data/${var.scriptfile}")}")}"
    }
  )

  depends_on = [azurerm_linux_virtual_machine.base[0]]
}

# Custom for work with both Windows and Linux Machine
# Notice: With that kind preference - The Command should be executed but
# With the Windows Machine: Cannot bypass sysprep with oobe for firtst time --> This extension will not execute
resource "azurerm_virtual_machine_extension" "bootstrap_command" {
  count                = ((var.checked_extension == true) && (var.type_of_extension_bootstrap == "command")) ? 1 : 0
  name                 = var.name_extension
  virtual_machine_id   = var.type_os_vm == "linux" ? azurerm_linux_virtual_machine.base[0].id : azurerm_windows_virtual_machine.base[0].id
  publisher            = var.type_os_vm == "linux" ? "Microsoft.Azure.Extensions" : "Microsoft.Compute"
  type                 = var.type_os_vm == "linux" ? "CustomScript" : "CustomScriptExtension"
  type_handler_version = var.type_os_vm == "linux" ? "2.0" : "1.10"
  settings = var.type_os_vm == "linux" ? jsonencode({
    "commandToExecute" : "${var.commandToExecute}"
    }) : jsonencode({
    "commandToExecute" : "powershell.exe -ExecutionPolicy Bypass -File C:\\Users\\startup.ps1"
  })

  depends_on = [azurerm_linux_virtual_machine.base[0], azurerm_windows_virtual_machine.base[0]]
}

So maybe it is super complicated because i custom that one for both Win and Linux can applied, so it need and can be complex with 100%

. But it have some feature you need to concern about that like:

• You can put your customize what image which you want to user for vm both Win and Linux is setup. Find out at dynamic "source_image_reference" that will decied you to choose what image source you want customize or raw.
• With Linux machine, it will help you have way to additional custom user_data which better way to running the initialize for first time setup this VM. Read more in this –> link. With Window, I confess this to harding for setup that, so it not support for window vm and you need to find out way to bypass this and Window part i will break out.
• resource "azurerm_virtual_machine_extension" this kind of thing can be cool with Azure VM, it can help you run extend script after init setup the vm. Pretty cool but it only work with the single instance don't try with scale set because scaleset itself intergration that one, go this link for know more . Especially, It just one again working only with Linux machine
• And another one it not only work with script, it can run command if you want by resource "azurerm_virtual_machine_extension" "bootstrap_command"

So after you got all that one we will go to detail of agent pools what it need

azure-agent/basepool/main.tf

resource "azurerm_resource_group" "basepool" {
  name     = var.pool_name
  location = var.location
  tags     = var.tag
}

resource "random_id" "basepool" {
  byte_length = 7
}

module "network" {
  source                          = "../../modules/network"
  resource_group_name             = azurerm_resource_group.basepool.name
  location                        = azurerm_resource_group.basepool.location
  tag                             = var.tag
  address_space                   = var.address_space
  address_prefixes_agent_subnet   = var.address_prefixes_agent_subnet
  address_prefixes_bastion_subnet = var.address_prefixes_bastion_subnet
  private_ip_address_allocation   = var.private_ip_address_allocation
  bastion_host_checked            = var.bastion_host_checked
  pool_name                       = var.pool_name
  type_os_vm                      = "linux"
  checked_scaleset                = true
}

module "vmss" {
  source = "../../modules/vm"

  # Choice Type of OS and Azure Resource
  type_os_vm       = "linux"
  checked_scaleset = true
  pool_name        = var.pool_name

  # To Provisioning VMSS
  resource_group_name           = azurerm_resource_group.basepool.name
  location                      = var.location
  admin_username                = var.admin_username
  admin_name_for_ssh            = var.admin_name_for_ssh
  agent_vm_size                 = var.agent_vm_size
  vmss_instances                = var.vmss_instances
  subnet_id                     = module.network.base_subnet_id
  agent_vm_caching              = var.agent_vm_caching
  agent_vm_storage_account_type = var.agent_vm_storage_account_type
  tag                           = var.tag
  nic_id                        = module.network.nic_id
  configuration_vm              = var.configuration_vm
  image_id                      = null

  # To Provisioning Agent and Extension
  pool_folder   = "basepool"
  agent_version = var.agent_version
  url_org       = var.url_org
  auth_type     = var.auth_type
  azure_pat     = var.azure_pat
  agent_name    = "Agent${random_id.basepool.dec}"
  workdir       = var.workdir
  tagagent      = random_id.basepool.dec

  depends_on = [module.network]
}

module "bastion_host" {
  count                = var.bastion_host_checked == true ? 1 : 0
  source               = "../../modules/bastion_host"
  location             = azurerm_resource_group.basepool.location
  resource_group_name  = azurerm_resource_group.basepool.name
  subnet_id            = module.network.bastionhost_subnet_id
  public_ip_address_id = module.network.public_ip_id
  pool_name            = var.pool_name
  depends_on           = [module.network]
}

With that one you include all things from what resource, define you custom varible and choose the what type you want to deploy this to Azure. Linux can be choice in here type_os_vm = "linux" and scale set version can be choice by checked_scaleset = true. So go for and read it to understand that said

.

But one more things you need to put the user-data script with template style and call that via template func of terraform. Dockerfile can be

#!/bin/bash

sudo apt update

sudo apt upgrade -y

sudo apt install -y docker-compose \
    curl \
    git \
    tar \
    jq \
    pass \
    gnupg2

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

## Bypass docker need root to execute
sudo chmod 666 /var/run/docker.sock

## Create ARG variable
VERSION="${version}"
URL_TFS="${url}"
AUTH_TYPE="${auth}"
TOKEN_AUTH="${token}"
POOL_AGENT="${pool}"
NAME_AGENT="${agent}"
TAG_AGENT="${tagagent}"
WORKDIR="${workdir}"

## Create dockerfile
cat <<EOF >/tmp/dockerfile
FROM debian:bullseye-slim

## The flag need to setup in env for bypass the check root in ./config.sh
ENV AGENT_ALLOW_RUNASROOT="1"

ENV AZP_AGENT_DOWNGRADE_DISABLED=true

WORKDIR /azp

RUN apt update && \
    apt install -y pass \
    gnupg2 \
    curl \ 
    git \
    jq \
    tar \
    unzip

RUN curl https://vstsagentpackage.azureedge.net/agent/"$VERSION"/vsts-agent-linux-x64-"$VERSION".tar.gz --output /tmp/download.tar.gz

RUN tar zxvf /tmp/download.tar.gz && \
    bash -c "bin/installdependencies.sh" && \
    ./config.sh --unattended --url "$URL_TFS" --auth "$AUTH_TYPE" --token "$TOKEN_AUTH" --pool "$POOL_AGENT" --agent "$NAME_AGENT" --work "$WORKDIR"

ENTRYPOINT [ "./run.sh" ]
EOF

# Move into directory && Create the image docker agent
cd /tmp && docker build -t agent:"$TAG_AGENT" -f dockerfile .

# Run agent with docker
docker run -d --name "$NAME_AGENT" agent:"$TAG_AGENT"

• All things is prepare and you are ready for deploy, use terraform CLI with workflow to applied it
  
• Optional when you choose using bastion host, your SSH Public Key is stored inside the tfstate of you. Go to and copy that for use connection to VM via that.
• And after all that, you machine will go on online inside your agent-pools and ready for connection via pipeline

Window

• With the window machine, i think it specially than with Linux. Because it is harder setup than linux, Close and Opensource that is different between them.
• And on the machine, it not open and set SSH for you connect to, instead that you need to get acquainted with winrm and rdp for work with windows
  • WinRM: IDK on the normal Window version like 10, 11 have it or not but on the Window Server especially 2019 version, all that one need to work with that. More detail about this protocol via this Link
  • RDP: This is protocol which help you connect with you Win VM and remote that via UI, this protocol work good but it not help me run any script via that, so poor for setting up anything.
• But not stop that, Window is have specify things that you not understand, It will obtain you go setup the machine with manual for first running time, some thing call OOBE. So you need to bypass that with third party, it is not recommended. Do not use this for production, experiment is recommended and go for this Link and this Link
• So go to detail that on what i am doing with Window Machine and how can i bypass them. On my window we will use construct terraform like this above and with windows machine i will setup automation test agent for C# project with .netcore, azure, vstest.
  Prequirement:

1. The agent should be able to use Windows VM because some reason:

   • The AutomationTest need Browser with GUI Support –> Chrome needs to be installed on VM (Both Windows and Linux can be but Windows will be easily)
   • The AutomationTest need the tools working with Powershell specific like AzureRM, Az, Azure and core of those things is AzurePS (Azure PowerShell convenient for setting up on Windows VM)
   • The Flexible PowerShell version can effectively for installation progess –> Have different between V5.0 and V7.0 of PowerShell
   • Setup dotnet core 7.0 (Both Windows and Linux can be installed but work with greate performs better in Windows than another OS)
   • Something tool like vstest.exe (Use in the Pipeline) –> Must be used with Windows VM for sure
   • Scripting use too much about TaskScheduler, Agent and another System lib like DLL file for work with.
     
     

2. Using Packer for packaging whole thing configuration. So why we need install packer with manual-install

3. Pipelines will need to setup Terraform for provisioning Agent and create some require for creating the agent. Those thing will reference to

4. Tools will use for Remote Connection to VM or Using the BastionHost (Need to configure to run sysprep.exe progress for Windows VM in Azure). With Windows you can using Remote Desktop or Linux can use Remmina to executing this one.

Features:

1. First of all, I will introduce some feature about template AzureVM Agent for configuration Which use for running the pipeline. Go and check on step one for more detail. This provisioning will design for using with template style which will help you deploy with different versions of AzureVM and you can use it with create a folder and import module like Using Env Terraform Folder.
2. The provisioning will combine two step with first step will run packer for packaging whole configuration for Windows VM and second step will provisioning the VM with image which build from first step.

Create a new AutomationTest Agent

1. Create a packer for packaging configuration (This one will help you bypass OOBE when setup windows machine)

• Go for packer session for more detail about that.
• Create a packer folder with construct
  • First, The constructure of packer, Packer is the tools of Hashicorp who design Terraform, so it will use the HCL language for doing the job. But different Terraform, you can flexible for split the folder into modules and env. But Packer will not be able to doing like that, You need add variable, local, packer and new thing like source and build block into one files, format of file will be <Name-you-want>.pkr.hcl, more details about packer-hcl. And the another thing call variables file for configuration flexibility variable for pkr.hcl file, format of file will be <Name-you-want>.pkrvars.hcl. But also you can use alternative styles of packer file in json format, more details about packer-json. Example of packer like:
    ​​​​​​​​Packer Folder:
    ​​​​​​​​    .
    ​​​​​​​​        |-variables.pkrvars.hcl
    ​​​​​​​​        |-automationtest.pkr.hcl
    ​​​​​​​​        |-data
    ​​​​​​​​        | |-setup-automation-agent.tpl
    ​​​​​​​​
    ​​​​​​​​*.pkr.hcl:
    ​​​​​​​​    variable "<name-of-variable>" {
    ​​​​​​​​        <Define variable need you parameterize for file>
    ​​​​​​​​        Ex:
    ​​​​​​​​            type        = string
    ​​​​​​​​            description = "Which Image Publisher to use for provisioning Image"
    ​​​​​​​​            default     = "MicrosoftWindowsServer"
    ​​​​​​​​    }
    
    ​​​​​​​​    source "provider" "<name-of-source>" {
    ​​​​​​​​        <Define Information VM Image you want to build>
    ​​​​​​​​        Ex:
    ​​​​​​​​            image_publisher                   = var.image_publisher
    ​​​​​​​​    }
    
    ​​​​​​​​    build {
    ​​​​​​​​        sources = [<List-source-block>]
    ​​​​​​​​        Ex:
    ​​​​​​​​            sources = ["sources.azure-arm.windows-machine"]
    
    ​​​​​​​​        provisioner "<tool-for-configuration-VM>" {
    ​​​​​​​​            <Configuration for VM like run setup, install tool, ...>
    ​​​​​​​​        Ex:
    ​​​​​​​​            inline = [local.script_content]
    ​​​​​​​​        }
    ​​​​​​​​    }
    
    ​​​​​​​​*.pkrvars.hcl
    ​​​​​​​​    image_publisher = "Cannonical"
    

  • Second for working with packer, you need to understand the provider which will work your cloud like Azure - AzureARM, AzureDTL or AWS - Instance, EBS. More detail about the provider can be found here: https://developer.hashicorp.com/packer/plugins
  • Third, on the building progress if you need to configure more thing for your VM, you will consider for using the provisioner in build block and it will connect to the VM with your what type of connection you configure on source block. With AzureARM, SSH will use for LinuxVM and WinRM will use for WindowsVM. For more details and choose for right provider, please see the documentation: https://developer.hashicorp.com/packer/docs/provisioners

2. All above will be scenarios, The results of the packer process will be like your what you configured. So for executing the packer process, you need to run the packer workflow for running the packer, more details please see the documentation. Before run the packer, you need configuration Authentication for use to communication cloud or what environment you want to build Image, Must be contribute role at lease which can perform create the resource on that platform.After authentication succeed, There are three steps which important when use packer.

   • init: used to download Packer plugin binaries
   ​​​​    Location: Your packer folder location
   
   ​​​​    Command:
   ​​​​        packer init .
   

   • validate: used to validate the syntax and configuration of a template
   ​​​​    Location: Your packer folder location
   
   ​​​​    Command:
   ​​​​        packer validate -var-file <varibles-file> <main-packer-file>
   

   • build: takes a template and runs all the builds within it in order to generate a set of artifacts
   ​​​​    Location
   
   ​​​​    Command:
   ​​​​        packer build -var-file <varibles-file> <main-packer-file>
   

   
   But take a little time for about my script to setup Window vm via winrm and powershell:

# Initialized parameter for the script
$localTempDir = $env:TEMP
$chromeInstaller = "ChromeInstaller.exe"
$Process2Monitor = "ChromeInstaller"
$zipfileDownload = "vsts-agent-win-x64-3.220.5.zip"
$powershellInstaller = "dotnet-install.ps1"

# Initialized parameter for agent specific
$URL_TFS = "${url}"
$AUTH_TYPE = "${auth}"
$TOKEN_AUTH = "${token}"
$POOL_AGENT = "${pool}"
$NAME_AGENT = "${agent}"
$WORKDIR = "${workdir}"

# Export environment variables for validation
[System.Environment]::SetEnvironmentVariable('AGENT_ALLOW_RUNASROOT', 1)
[System.Environment]::SetEnvironmentVariable('AZP_AGENT_DOWNGRADE_DISABLED', $true)

# Install Chrome 
(New-Object System.Net.WebClient).DownloadFile('http://dl.google.com/chrome/install/375.126/chrome_installer.exe', "$LocalTempDir\$ChromeInstaller")
& "$localTempDir\$ChromeInstaller" /silent /install
Do {
    $ProcessesFound = Get-Process | Where-Object { $Process2Monitor -contains $_.Name } | Select-Object -ExpandProperty Name
    If ($ProcessesFound) { "Still running: $($ProcessesFound -join ', ')" | Write-Host; Start-Sleep -Seconds 2 }
    else { Remove-Item "$LocalTempDir\$ChromeInstaller" -ErrorAction SilentlyContinue -Verbose } 
} Until (!$ProcessesFound)

# Install dotnet
mkdir "$env:CommonProgramFiles\dotnet"
Invoke-WebRequest "https://dot.net/v1/$powershellInstaller" -OutFile "$localTempDir\$powershellInstaller"
& "$localTempDir\$powershellInstaller" -Runtime dotnet -Version 7.0.9 -InstallDir "$env:ProgramFiles\dotnet"
dotnet --info
& "$localTempDir\$powershellInstaller" -Version 7.0.306 -Channel 7.0.9 -InstallDir "$env:ProgramFiles\dotnet"
dotnet --version
setx PATH "$env:path;$env:ProgramFiles\dotnet" -m

# Install Azure-CLI and Azureps (Availability for pipelines)
# Install Azure-CLI (Not needed but for sure)
Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi; Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'; Remove-Item .\AzureCLI.msi
# Install Azureps
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
Install-Module -Name Azure, Azure.Storage, AzureRM -Repository PSGallery -AllowClobber -Force
(Get-Module -ListAvailable | Where-Object { $_.Name -eq 'Azure' }) ` | Select-Object Version, Name, Author, PowerShellVersion  | Format-List;
(Get-Module -ListAvailable | Where-Object { $_.Name -eq 'AzureRM' }) ` | Select-Object Version, Name, Author, PowerShellVersion  | Format-List;

# Connect to Pool and Create a new Agent
Set-Location C:\
mkdir agent
Set-Location agent
(New-Object System.Net.WebClient).DownloadFile('https://vstsagentpackage.azureedge.net/agent/3.220.5/vsts-agent-win-x64-3.220.5.zip', "$LocalTempDir\$zipfileDownload")
Add-Type -AssemblyName System.IO.Compression.FileSystem ; [System.IO.Compression.ZipFile]::ExtractToDirectory("$LocalTempDir\$zipfileDownload", "$PWD")
.\config.cmd --unattended --url "$URL_TFS" --auth "$AUTH_TYPE" --token "$TOKEN_AUTH" --pool "$POOL_AGENT" --agent "$NAME_AGENT" --work "$WORKDIR"

# Create the EOF file for saving the file for later execution
@"
# Create the the schedule job
Register-ScheduledJob -Trigger (New-JobTrigger -AtStartup -RandomDelay 00:00:30) -Name agentRun -ScriptBlock { C:\agent\run.cmd }
# Restart computer for the first time
Restart-Computer
"@ | Out-File "C:\Users\startup.ps1"

# Check the file exists
Get-Content -Path "C:\Users\startup.ps1"

# Step to setup the Windows guest agent paricipation in sysprep process - (Must be exist for complete build Image for Windows Azure VM)
while ((Get-Service RdAgent).Status -ne 'Running') { 
    Start-Sleep -s 5 
}
while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { 
    Start-Sleep -s 5 
}
& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit
while ($true) { 
    $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select-Object ImageState
    if ($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { 
        Write-Output $imageState.ImageState
        Start-Sleep -s 10 
    } 
    else { 
        break 
    } 
}

You need to understand that why i am need to bypass OOBE, if don't have setup OOBE, your provisioning progress of Window will break and not succesfull.

3. After packer and you have image, you need to go for create VM with your customize image by terraform, main.tf here i come

resource "azurerm_resource_group" "basepool" {
  name     = var.pool_name
  location = var.location
  tags     = var.tag
}

resource "random_id" "basepool" {
  byte_length = 7
}

module "network" {
  source                          = "../../modules/network"
  resource_group_name             = azurerm_resource_group.basepool.name
  location                        = azurerm_resource_group.basepool.location
  tag                             = var.tag
  address_space                   = var.address_space
  address_prefixes_agent_subnet   = var.address_prefixes_agent_subnet
  address_prefixes_bastion_subnet = var.address_prefixes_bastion_subnet
  private_ip_address_allocation   = var.private_ip_address_allocation
  bastion_host_checked            = var.bastion_host_checked
  pool_name                       = var.pool_name
  type_os_vm                      = "linux"
  checked_scaleset                = true
}

module "vmss" {
  source = "../../modules/vm"

  # Choice Type of OS and Azure Resource
  type_os_vm       = "linux"
  checked_scaleset = true
  pool_name        = var.pool_name

  # To Provisioning VMSS
  resource_group_name           = azurerm_resource_group.basepool.name
  location                      = var.location
  admin_username                = var.admin_username
  admin_name_for_ssh            = var.admin_name_for_ssh
  agent_vm_size                 = var.agent_vm_size
  vmss_instances                = var.vmss_instances
  subnet_id                     = module.network.base_subnet_id
  agent_vm_caching              = var.agent_vm_caching
  agent_vm_storage_account_type = var.agent_vm_storage_account_type
  tag                           = var.tag
  nic_id                        = module.network.nic_id
  configuration_vm              = var.configuration_vm
  image_id                      = null

  # To Provisioning Agent and Extension
  pool_folder   = "basepool"
  agent_version = var.agent_version
  url_org       = var.url_org
  auth_type     = var.auth_type
  azure_pat     = var.azure_pat
  agent_name    = "Agent${random_id.basepool.dec}"
  workdir       = var.workdir
  tagagent      = random_id.basepool.dec

  depends_on = [module.network]
}

module "bastion_host" {
  count                = var.bastion_host_checked == true ? 1 : 0
  source               = "../../modules/bastion_host"
  location             = azurerm_resource_group.basepool.location
  resource_group_name  = azurerm_resource_group.basepool.name
  subnet_id            = module.network.bastionhost_subnet_id
  public_ip_address_id = module.network.public_ip_id
  pool_name            = var.pool_name
  depends_on           = [module.network]
}

Some variable you need configuration for Terraform working (Consider to change for make sure right configuration):

Name	Description	Type	Default	Required
pool_name	Name of agent pool	string	N/A	True
location	Location of resource	string	southeastasia	False
use_image	Conditionally for optional use Packer Image or Not	bool	True	False
image_resource_group	RG where storage Packer Image	string	"PackerVMImage"	False
image_name	RG where storage Packer Image	string	N/A	True
azure_pat	Azure Personal access token	string	N/A	True
url_org	URL of organization where give access for pool	string	N/A	True
agent_vm_size	Size of VM for agent	string	Standard_B1ms	False
workdir	work directory for pool	string	usr/local/agent_work	False

The Terraform of Agent which build on Template so make sure the type of variable is needed for doing right job.

4. Remote Access into VM (IMPORTANT STEP !! PLEASE DO THIS STEP MANUALLY FOR HELP RUNNING SYSPREP)

• Like i said when you provision Win VM, it have some different, Go for different about The Consistency of Windows VM in Azure Cloud:

  1. When you build the VM you need verify that your Image need to be have sysprep.exe for wipe anything before packing into VM. More information in documentation: https://learn.microsoft.com/en-us/azure/virtual-machines/windows/build-image-with-packer
  2. Windows VM have different type to connection in Azure, It will not configuration SSH for first thing provisioning so you cannot performance that because that will not support but Windows have another way to connection like RDP and WinRM. With RDP Protocol, you can access and get the GUI of Windows Machine but it will not like ssh you cannot perform script via RDP. So windows exchange and use winrm is replacing solution but in the basic image Windows will not have permission to execution that protocol, you need configuration it about packer or with documentation https://learn.microsoft.com/en-us/windows/win32/winrm/installation-and-configuration-for-windows-remote-management
  3. So may be it is possible but first of time, You connect to Windows it will need you perform the sysprep of machine, so RDP, WinRM or ExtensionScript cannot help us doing the Schedule Task so that why you need doing Manual on that step

• So you can perform sysprep for first setup with RDP through port 3389 with windows you can using Remote Desktop or Linux can use Remmina to executing.

• Maybe have more way can bypass the sysprep but it not recommended because this can cause problems with Windows (Like i said on the head of this). So that step it need to be obligatory

• After that you first time the agent configuration is setup inside the Windows Machine on Packer Step. So things you need to do first is go to Users folder at location C:\Users\ for doing Powershell Script startup.ps1. That kind is just a thinks for use setup the startup job via schedule job - Tool of window.

• After the machine restart about deplay 30 second the agent will start

The last of the last, Like i promise you will create bastion host free with low cost to operation

• This is things about network, so you have different network between this resources and that resources. Find the way to connect them that is solution
  
• So you just need about take care one more thing how can peer the network with themand you will custom your machine to bastionhost. Go detail on this Link. This is more way (VPN, Network peering, …) but network peering is easily to setup via azure
  1. First you need to decide what network you want to peering (Remember: Your VPC or Virtual Network need to split in different range, if similar you can do in with that) and go to peering tab
     
  2. Choose add option and fill the name what you want, choose what rule you applied you this peering, choose subscription and what virtual network other you want. Save and go live it
     
  3. Go to the virtual machine and try to ping go through another one, with Windows Machine you need to turn off the firewall or set rule before ping and Linux is not need. That quite pretty easily
     

Conclusion

• That all of things which i want to share with you about setup Linux and Windows agent for purpose setting Agent for Azure Pipeline. Hope you solve with hard things and happily if read it to the end
• With that one you can do anything pipeline with specify repo like github, azure devops git, gitlab and what ever where you want just need a token to access
• With the terraform, you will learn how to setup template of Win and Linux for recycling code for mutiple purpose
• Packer is really good for optimize anything, this will help you compress anything into one and optimize the time you give for running whole this for multiple time.
• OOBE is best, Window can help you know about more and discover them can help you learn to the best, Bypass OOBE is popular things i find in DevOps and SysOps forum
• Azure is the supicious thing which some resource with high complex, but probally it have logic with tough and tightness. So you need to learning more to understand this loud
• The network peering is not hard like you thought. So try that and optimize the cost for give you equivalent value.
  

Have a good day for setup and configure. Bye and see you on next blog