Sunday, March 22, 2020

Terraform - count vs for_each

Terraform Variables  -  Input variables   


Count:

Before Terraform 0.12.6, the only way to create multiple instances of the same resource was to use a count parameter. One of the problems with this approach  is ordering. Count is maintaining the  array numeric index (list) to perform it's operations.If  there is a change in the order, terraform wants to destroy/re-create that object.

Code snippet has been given below to explain the difference between count and for_each. Given snippet has been taken from block volume provisioning & attachment module.

Phase 1: Provision the block volumes  

Below code will provision three block volumes("MyVolume1","MyVolume2","jay")  and attach the same to the defined compute instance. 

Variables.tf
variable "block_display_name" {
type = "list"
default = ["MyVolume1","MyVolume2","jay"]
}
variable "block_size" {
type = "list"
default = ["50","60","80"]
}

block.tf

resource "oci_core_volume" "gol_blockvolume" {
  count =  "${var.vol_count}"
   availability_domain = "${data.oci_identity_availability_domain.ad.name}"
  compartment_id      = "${var.compartment_id}"
  display_name        = "${var.block_display_name[count.index]}"
  size_in_gbs         = "${var.block_size[count.index]}"
 }

resource "oci_core_volume_attachment" "gol_attachment" {
  count =  "${var.vol_count}"
  depends_on = ["oci_core_volume.gol_blockvolume"]
  attachment_type = "iscsi"
  instance_id     = "${data.oci_core_instances.gol_instances.instances.*.id[0]}"
  volume_id       = "${oci_core_volume.gol_blockvolume.*.id[count.index]}"
}

Let us run terraform plan to review  the resource actions.


Plan output:
 
 [oracle@terraform1 storage]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:
# module.block1.oci_core_volume.gol_blockvolume[0] will be created
  + resource "oci_core_volume" "gol_blockvolume" {
      + availability_domain = "tGNj:US-ASHBURN-AD-1"
      + backup_policy_id    = (known after apply)
      + compartment_id      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxxxxxxxxxx"
      + defined_tags        = (known after apply)
      + display_name        = "MyVolume1"
      + freeform_tags       = (known after apply)
      + id                  = (known after apply)
      + is_hydrated         = (known after apply)
      + kms_key_id          = (known after apply)
      + size_in_gbs         = "50"
      + size_in_mbs         = (known after apply)
      + state               = (known after apply)
      + system_tags         = (known after apply)
      + time_created        = (known after apply)
      + volume_backup_id    = (known after apply)
      + volume_group_id     = (known after apply)

      + source_details {

          + id   = (known after apply)
          + type = (known after apply)
        }
    }

  # module.block1.oci_core_volume.gol_blockvolume[1] will be created

  + resource "oci_core_volume" "gol_blockvolume" {
      + availability_domain = "tGNj:US-ASHBURN-AD-1"
      + backup_policy_id    = (known after apply)
      + compartment_id      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxxxxxxxxxx"
      + defined_tags        = (known after apply)
      + display_name        = "MyVolume2"
      + freeform_tags       = (known after apply)
      + id                  = (known after apply)
      + is_hydrated         = (known after apply)
      + kms_key_id          = (known after apply)
      + size_in_gbs         = "60"
      + size_in_mbs         = (known after apply)
      + state               = (known after apply)
      + system_tags         = (known after apply)
      + time_created        = (known after apply)
      + volume_backup_id    = (known after apply)
      + volume_group_id     = (known after apply)

      + source_details {

          + id   = (known after apply)
          + type = (known after apply)
        }
    }

  # module.block1.oci_core_volume.gol_blockvolume[2] will be created

  + resource "oci_core_volume" "gol_blockvolume" {
      + availability_domain = "tGNj:US-ASHBURN-AD-1"
      + backup_policy_id    = (known after apply)
      + compartment_id      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxxxxxxxxxx"
      + defined_tags        = (known after apply)
      + display_name        = "jay"
      + freeform_tags       = (known after apply)
      + id                  = (known after apply)
      + is_hydrated         = (known after apply)
      + kms_key_id          = (known after apply)
      + size_in_gbs         = "80"
      + size_in_mbs         = (known after apply)
      + state               = (known after apply)
      + system_tags         = (known after apply)
      + time_created        = (known after apply)
      + volume_backup_id    = (known after apply)
      + volume_group_id     = (known after apply)

      + source_details {

          + id   = (known after apply)
          + type = (known after apply)
        }
    }

  # module.block1.oci_core_volume_attachment.gol_attachment[0] will be created

  + resource "oci_core_volume_attachment" "gol_attachment" {
      + attachment_type                     = "iscsi"
      + availability_domain                 = (known after apply)
      + chap_secret                         = (known after apply)
      + chap_username                       = (known after apply)
      + compartment_id                      = (known after apply)
      + device                              = (known after apply)
      + display_name                        = (known after apply)
      + id                                  = (known after apply)
      + instance_id                         = "ocid1.instance.oc1.iad.xxxxxxxxxxxxxxxxxxxxxxx"
      + ipv4                                = (known after apply)
      + iqn                                 = (known after apply)
      + is_pv_encryption_in_transit_enabled = (known after apply)
      + is_read_only                        = (known after apply)
      + port                                = (known after apply)
      + state                               = (known after apply)
      + time_created                        = (known after apply)
      + use_chap                            = (known after apply)
      + volume_id                           = (known after apply)
    }

  # module.block1.oci_core_volume_attachment.gol_attachment[1] will be created

  + resource "oci_core_volume_attachment" "gol_attachment" {
      + attachment_type                     = "iscsi"
      + availability_domain                 = (known after apply)
      + chap_secret                         = (known after apply)
      + chap_username                       = (known after apply)
      + compartment_id                      = (known after apply)
      + device                              = (known after apply)
      + display_name                        = (known after apply)
      + id                                  = (known after apply)
      + instance_id                         = "ocid1.instance.oc1.iad.xxxxxxxxxxxxxxxxxxxxxxx"
      + ipv4                                = (known after apply)
      + iqn                                 = (known after apply)
      + is_pv_encryption_in_transit_enabled = (known after apply)
      + is_read_only                        = (known after apply)
      + port                                = (known after apply)
      + state                               = (known after apply)
      + time_created                        = (known after apply)
      + use_chap                            = (known after apply)
      + volume_id                           = (known after apply)
    }

  # module.block1.oci_core_volume_attachment.gol_attachment[2] will be created

  + resource "oci_core_volume_attachment" "gol_attachment" {
      + attachment_type                     = "iscsi"
      + availability_domain                 = (known after apply)
      + chap_secret                         = (known after apply)
      + chap_username                       = (known after apply)
      + compartment_id                      = (known after apply)
      + device                              = (known after apply)
      + display_name                        = (known after apply)
      + id                                  = (known after apply)
      + instance_id                         = "ocid1.instance.oc1.iad.xxxxxxxxxxxxxxxxxxxxxxx"
      + ipv4                                = (known after apply)
      + iqn                                 = (known after apply)
      + is_pv_encryption_in_transit_enabled = (known after apply)
      + is_read_only                        = (known after apply)
      + port                                = (known after apply)
      + state                               = (known after apply)
      + time_created                        = (known after apply)
      + use_chap                            = (known after apply)
      + volume_id                           = (known after apply)
    }

Plan: 6 to add, 0 to change, 0 to destroy.


After terraform apply, we got our blockvolumes provisioned &  attached to the requested compute

 
 Phase 2: Remove the MyVolume2  

Let us remove the MyVolume2 from variables.tf  & run terraform plan.

Variables.tf

variable "block_display_name" {
type = "list"
default = ["MyVolume1","jay"]
}
variable "block_size" {
type = "list"
default = ["50","80"]
}


plan output:


[oracle@terraform1 storage]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

  ~ update in-place
  - destroy


Terraform will perform the following actions:

  # module.block1.oci_core_volume.gol_blockvolume[1] will be updated in-place
  ~ resource "oci_core_volume" "gol_blockvolume" {
        availability_domain = "tGNj:US-ASHBURN-AD-1"
        compartment_id      = "ocid1.compartment.oc1.xxxxxxxxxxxxxxx"
        defined_tags        = {
            "Oracle-Tags.CreatedBy" = "xxxxxxxx@gmail.com"
            "Oracle-Tags.CreatedOn" = "2020-03-21T12:03:25.030Z"
        }
      ~ display_name        = "MyVolume2" -> "jay"
        freeform_tags       = {}
        id                  = "ocid1.volume.oc1.iad.abuwcljs5gea4diqvucb65ckwmups27of3ixxjkgblvylulpbqev3jrwzlja"
        is_hydrated         = true
      ~ size_in_gbs         = "60" -> "80"
        size_in_mbs         = "61440"
        state               = "AVAILABLE"
        system_tags         = {}
        time_created        = "2020-03-21 12:03:25.652 +0000 UTC"
    }

  # module.block1.oci_core_volume.gol_blockvolume[2] will be destroyed
  - resource "oci_core_volume" "gol_blockvolume" {
      - availability_domain = "tGNj:US-ASHBURN-AD-1" -> null
      - compartment_id      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxx"" -> null
      - defined_tags        = {
          - "Oracle-Tags.CreatedBy" = "xxxxxxxx@gmail.com"
          - "Oracle-Tags.CreatedOn" = "2020-03-21T12:03:25.032Z"
        } -> null
      - display_name        = "jay" -> null
      - freeform_tags       = {} -> null
      - id                  = "ocid1.volume.oc1.iad.xxxxxxxxxxxxxxx"" -> null
      - is_hydrated         = true -> null
      - size_in_gbs         = "80" -> null
      - size_in_mbs         = "81920" -> null
      - state               = "AVAILABLE" -> null
      - system_tags         = {} -> null
      - time_created        = "2020-03-21 12:03:25.251 +0000 UTC" -> null
    }

  # module.block1.oci_core_volume_attachment.gol_attachment[2] will be destroyed
  - resource "oci_core_volume_attachment" "gol_attachment" {
      - attachment_type                     = "iscsi" -> null
      - availability_domain                 = "tGNj:US-ASHBURN-AD-1" -> null
      - compartment_id                      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxx"" -> null
      - display_name                        = "volumeattachment20200321121512" -> null
      - id                                  = "ocid1.volumeattachment.oc1.iad.xxxxxxxxxxxxxxx"" -> null
      - instance_id                         = "ocid1.instance.oc1.iad.xxxxxxxxxxxxxxx"" -> null
      - ipv4                                = "169.254.2.3" -> null
      - iqn                                 = "iqn.2015-12.com.oracleiaas:4d2085c9-" -> null
      - is_pv_encryption_in_transit_enabled = false -> null
      - is_read_only                        = false -> null
      - port                                = 3260 -> null
      - state                               = "ATTACHED" -> null
      - time_created                        = "2020-03-21 12:15:12.62 +0000 UTC" -> null
      - volume_id                           = "ocid1.volume.oc1.iad.xxxxxxxxxxxxxxx"" -> null
    }

Plan: 0 to add, 1 to change, 2 to destroy.


Expectation 

Remove the block volume "MyVolume2"  & Keep the block volumes  Myvolume1 & jay untouched.

Reality

It destroys compute instance attachment & block volume of  jay (AND) Renames Myvolume2 as Jay  --- This is a DISASTER..

How it works
  
As we mentioned earlier, count is maintaining the  array numeric index (list) to perform it's operations. For example,Please find the list representation below.

Array representation of block volumes:
module.block1.oci_core_volume.gol_blockvolume[0] = MyVolume1
module.block1.oci_core_volume.gol_blockvolume[1] = MyVolume2
module.block1.oci_core_volume.gol_blockvolume[2] =jay

When you remove an item from the middle of an array, all the items after that shift back by one.

module.block1.oci_core_volume.gol_blockvolume[0] = MyVolume1
module.block1.oci_core_volume.gol_blockvolume[1] = jay

As terraform count works  by considering index as resource identity, it  renames the array[1] from Myvolume2 to jay & destroy module.block1.oci_core_volume.gol_blockvolume[2] : jay.  

Let us stop at this stage & don't run terraform apply.

For_each

It takes a map / set as input and uses the key of a map as an index of instances of created resource. If we are using list datatype, we can use toset function to convert  list to maps.

  Phase 1: Provision the block volumes. 

Below code will provision three block volumes("MyVolume1","MyVolume2","jay") and attach the same to the compute instance mentioned.

 variables.tf:

variable "block_volume"{
type = "map"
default = {
MyVolume1 = "50"
MyVolume2 = "90"
jay       = "80"
}
}

block.tf

[oracle@terraform1 storage]$ more block_volume.tf
resource "oci_core_volume" "gol_blockvolume" {
  for_each             = var.block_volume
  availability_domain = "${data.oci_identity_availability_domain.ad.name}"
  compartment_id      = "${var.compartment_id}"
  display_name = each.key
  size_in_gbs  = each.value
}

resource "oci_core_volume_attachment" "gol_attachment" {
  for_each             =var.block_volume
  depends_on = ["oci_core_volume.gol_blockvolume"]
  attachment_type = "iscsi"
  instance_id     = "${data.oci_core_instances.gol_instances.instances.*.id[0]}"
  volume_id       = "${oci_core_volume.gol_blockvolume[each.key].id}"
}


After Apply:


Phase 2: Remove the MyVolume2  

Let us remove the MyVolume2 from variables.tf 


variable "block_volume"{
type = "map"
default = {
MyVolume1 = "50"
jay       = "80"

}
}


Now execute terraform plan.


[oracle@terraform1 storage]$ terraform plan
Refreshing Terraform state in-memory prior to plan...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # module.block1.oci_core_volume.gol_blockvolume["MyVolume2"] will be destroyed

  - resource "oci_core_volume" "gol_blockvolume" {
      - availability_domain = "tGNj:US-ASHBURN-AD-1" -> null
      - compartment_id      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxx"" -> null
      - defined_tags        = {
          - "Oracle-Tags.CreatedBy" = "xxxxxx@gmail.com"
          - "Oracle-Tags.CreatedOn" = "2020-03-21T14:57:59.208Z"
        } -> null
      - display_name        = "MyVolume2" -> null
      - freeform_tags       = {} -> null
      - id                  = "ocid1.volume.oc1.iad.xxxxxxxxxxxxxxx"" -> null
      - is_hydrated         = true -> null
      - size_in_gbs         = "90" -> null
      - size_in_mbs         = "92160" -> null
      - state               = "AVAILABLE" -> null
      - system_tags         = {} -> null
      - time_created        = "2020-03-21 14:57:59.648 +0000 UTC" -> null
    }

  # module.block1.oci_core_volume_attachment.gol_attachment["MyVolume2"] will be destroyed

  - resource "oci_core_volume_attachment" "gol_attachment" {
      - attachment_type                     = "iscsi" -> null
      - availability_domain                 = "tGNj:US-ASHBURN-AD-1" -> null
      - compartment_id                      = "ocid1.compartment.oc1..xxxxxxxxxxxxxxx"" -> null
      - display_name                        = "volumeattachment20200321145822" -> null
      - id                                  = "ocid1.volumeattachment.oc1.iad.xxxxxxxxxxxxxxx"" -> null
      - instance_id                         = "ocid1.instance.oc1.iad.xxxxxxxxxxxxxxx"" -> null
      - ipv4                                = "169.254.2.6" -> null
      - iqn                                 = "iqn.2015-12.com.oracleiaas:xxxxxxxxxxxxxxx"" -> null
      - is_pv_encryption_in_transit_enabled = false -> null
      - is_read_only                        = false -> null
      - port                                = 3260 -> null
      - state                               = "ATTACHED" -> null
      - time_created                        = "2020-03-21 14:58:22.751 +0000 UTC" -> null
      - volume_id                           = "ocid1.volume.oc1.iad.xxxxxxxxxxxxxxx" -> null
    }

Plan: 0 to add, 0 to change, 2 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.


Great. It works as expected.

How it works:

for_each  is using the key of a map as an index  to perform it's operations. Terraform will identify each instance by the string key of the map element rather than by a numeric index,For example,Please find the representation below.

Array representation of block volumes:
module.block1.oci_core_volume.gol_blockvolume[MyVolume1] = 50
module.block1.oci_core_volume.gol_blockvolume[MyVolume2] =90
module.block1.oci_core_volume.gol_blockvolume[jay] =80

When you remove an item from the middle of an array,it just destroys the removed item.

module.block1.oci_core_volume.gol_blockvolume[MyVolume1] = 50
module.block1.oci_core_volume.gol_blockvolume[jay] =80


Ref: https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/

2 comments:

  1. You’d outstanding guidelines there. I did a search about the field and identified that very likely the majority will agree with your web page.
    Thanks for your post. This is excellent information.
    DevOps Training in Chennai

    DevOps Online Training in Chennai

    DevOps Training in Bangalore

    DevOps Training in Hyderabad

    DevOps Training in Coimbatore

    DevOps Training

    DevOps Online Training

    ReplyDelete
  2. Great Info, Thanks For Sharing , keep it up we are here to learn more

    Great! I like to share it with all my friends and hope they will also like this information.
    Power BI Training In Hyderabad
    Power BI Online Training
    Power BI Training
    Power BI Training Online

    ReplyDelete

How to Compile Forms , Reports & Custom.pll in R12.2

How to Compile Custom.pll   cd $AU_TOP/resource  cp CUSTOM.plx CUSTOM.plx_bkup  cp CUSTOM.pll CUSTOM.pll_bkup  frmcmp_batch module=CUSTOM.pl...