CODEWITHSUNDEEP
yamlyq
Raj
Raj

Reputation: 601

upgrading to yq v4 from v3 - merging YAML files

I have been using yq v3 to merge yaml files using the following command.

yq m <yaml_file> <yaml_file2> <yaml_file3>

The command works as outlined here

For example, let's say I have the following 3 yaml files.

training-lab-account.yml

account:
  alias: training-lab-account
  orgEmail: [email protected]
  snsSubscriptions:
    - [email protected]
network:
  vpcs:
    - cidr: 10.253.0.0/16
  snsSubscriptions:
    - [email protected]

account.yml

---
account:
  snsSubscriptions:
    - [email protected]
  orgEmail: REQUIRED
  orgAccountProvisioningEnabled: true
  groups:
    - name: SmileAdmins
      policies:
        - AdministratorAccess
    - name: SmileReadOnly
      policies:
        - ReadOnlyAccess

network.yml

---
network:
  snsSubscriptions:
    - [email protected]
  vpcs:
    - vpcName: Main
      amiPipeline: false
      accountVendingMachine: false
      flowlogs: true
      transitGw: false
      maxAzs: 3
      endpoints:
        gateway:
          - name: s3
          - name: dynamodb
        interface:
          - name: ssm

when I execute yq m training-lab-account.yml account.yml network.yml, I get the following output.

account:
  alias: training-lab-account
  orgEmail: [email protected]
  snsSubscriptions:
    - [email protected]
  orgAccountProvisioningEnabled: true
  groups:
    - name: SmileAdmins
      policies:
        - AdministratorAccess
    - name: SmileReadOnly
      policies:
        - ReadOnlyAccess
network:
  vpcs:
    - cidr: 10.253.0.0/16
      vpcName: Main
      amiPipeline: false
      accountVendingMachine: false
      flowlogs: true
      transitGw: false
      maxAzs: 3
      endpoints:
        gateway:
          - name: s3
          - name: dynamodb
        interface:
          - name: ssm
  snsSubscriptions:
    - [email protected]

Basically, account.yml and network.yml are merged into training-lab-account.yml without overwriting the content in training-lab-account.yml.

yq v4 doesn't support the "m" flag to merge anymore. Actually I feel like yq@4 is a brand new piece of software and is completely different from yq@3.

Long story short, I am trying to merge using eval-all as outlined here: yq eval-all --inplace 'select(fileIndex == 0) * select(fileIndex == 1)' f1.yml f2.yml

Unfortunately, this overwrites what's in f1.yml from f2.yml - which is not how merge behaves in yq@3.

Since yq@3 is deprecated, I need to upgrade to yq@4. I would appreciate any feedback on how to replicate the merge feature from yq@3 in yq@4.

Upvotes: 1

Views: 2710

Answers (2)

Pete
Pete

Reputation: 807

In your general example

yq eval-all --inplace 'select(fileIndex == 0) * select(fileIndex == 1)' f1.yml f2.yml

f2.yml overwrites all values that exist in f1.yml. This corresponds to yq v3 behavior of yq m -ix f1.yml f2.yml

If you want to keep values from f1.yml you can simply switch the order:

yq eval-all --inplace 'select(fileIndex == 1) * select(fileIndex == 0)' f1.yml f2.yml

Note, that we switch the order in the select command but keep the order of the given filenames. This ensures that f1.yml is the file that is rewritten in-place.

I am pretty sure, you can define the order in the select command in a way that your 3 file merge also works as you want it.

Upvotes: 0

flyx
flyx

Reputation: 39638

If you give --inplace, it writes the result to the first given file. Just drop that parameter.

Edit: To have the values in training-lab-account.yml take precedence over those in the other files, give it as last parameter. To merge the sequence values in vpcs:, use *d to enable deep array merging. Resulting command:

yq eval-all 'select(fileIndex == 0) *d select(fileIndex == 1) *d select(fileIndex == 2)' account.yml network.yml training-lab-account.yml

Upvotes: 2

Related Questions