Learning to Manipulate YAML on the Command Line with YQ

You are probably familiar with “jq”. Every developer or DevOps must have handled JSON data. “jq” is a lightweight and flexible command-line JSON processor akin to sed, awk, and grep. Those linux commands like sed, awk, and grep are awesome tools, but when you are working on formatted data like JSON or YAML, a private library that is developed for this format can be a better solution because you do not have to write complex linux command syntaxes.

“yq” is a lightweight and portable command-line YAML, JSON, and XML processor. “yq” uses “jq” as syntax but works with YAML files as well as JSON, XML, properties, CSV, and TSV. When you search for “yq”, you can find two repositories. But the “yq” tool we refer to in this article is mikefarah/yq, not “kislyuk/yq. When you try to install yq from the Ubuntu Snap repository, “mikefarah/yq” will be installed. The version needs to be "4.x" which is crucial. There are huge differences between “3.x” and “4.x”.

YAML files are so popular nowadays. One reason is that Kubernetes supports YAML files. YAML files are important because they offer several benefits, like being better human-readable than others. As a DevOps and Kubernetes guru, I can write a huge Yaml file with a simple IDE. Whatever, we need YAML file operations, so we need “yq”.

Before we get to the miracles of “yq”, let’s take a look at that below.

That is a simple YAML file ( your_yaml_file_1.yaml ):

staff:
 - name: "Batur"
   surname: "Orkun"
   city: "Ankara"
   street: "My street 1"
   number: 1
 - name: "Ebru"
   surname: "Orkun"
   city: "Antalya"
   street: "My street 2"
   number: 2

I want to pull out the city name of the staff member named Ebru.

“Antalya”

Using “awk”:

awk -F ': ' '/- name: "Ebru"/ {getline; getline; print $2}'  your_yaml_file_1.yaml | tr -d '"'

Using “yq”:

yq e '.staff[] | select(.name == "Ebru") | .city' your_yaml_file_1.yaml

You can easily see which one is smooth and understandable.

Maybe your YAML is a little more complex. ( your_yaml_file_2.yaml )

staff:
 - name: "Batur"
   surname: "Orkun"
   address:
      city: "Ankara"
      street: "My street 1"
      number: 1
 - name: "Ebru"
   surname: "Orkun"
   address:
      city: "Antalya"
      street: "My street 2"
      number: 2

Same request. I want to pull out the city name of the staff member named Ebru.

Using “awk”:

awk -F ': ' '/- name: "Ebru"/ {getline; while (getline) { if ($1 ~ /city/) { print $2; break; } }}' your_yaml_file_2.yaml

Using “yq”

yq eval '.staff[] | select(.name == "Ebru") | .address.city' your_yaml_file_2.yaml

You can see that we found the word "city" by regular expression while using awk. If we add another keyword to YAML, like “publicity”. “if regex” will find this line, and the awk command will return the wrong value. So our commands can be easily affected when the structure of the data changes.

How can you install “yq”? From Snap, Brew, or GitHub.

For Mac / OS X:

brew install yq

For Ubuntu/Debian:

snap install yq

For any Linux (64 bit):

wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
 chmod +x /usr/bin/yq

“yq” abandoned those subcommands like r and d and instead triggered all operations with yq eval in version “4.x”

yq eval [expression] [yaml_file1]… [flags]
yq e [expression] [yaml_file1]… [flags]
yq [expression] [yaml_file1]… [flags]

All of them are the same because eval is the default command. You can get or change any data in any condition from a YAML file. Let’s take a look at the basics.

That is the “fruits.yaml” file:

fruits:
  apple: red
  pear: green
  banana: yellow

If you wonder about the color of the pear:

yq '.fruits.pear' fruits.yml

Output is “green”.

If you want to change the value, use the "-i" parameter.

yq e '.fruits.apple = "verdant"' -i fruits.yml

After running the command, when you look at the file:

fruits:
  apple: verdant 
  pear: green
  banana: yellow

YAML files generally have array values. You must learn to work on array values. Let’s work on Kubernetes files.

apiVersion: v1
kind: Pod
metadata:
 name: test-pod
spec:
 containers:
   - name: test-container
     image: busybox
     env:
       — name: DB_CONNECT
         value: postgres://192.168.10.10:5432
       - name: SHARED_PATH
         value: /var/nfs

We need the IP of the Postgresql database from that YAML above.

yq ".spec.containers[0].env[0].value" pod.yaml

Output is "postgres://192.168.10.10:5432”.

We can use “awk” to parse the IP address.

yq ".spec.containers[0].env[0].value" pod.yaml | awk -F// '{split($2, a, ":"); print a[1]}'

We accessed the first value in the env array. If we change the order, we get the wrong value. So maybe we can get the value by selecting the name.

yq '.spec.containers[0].env[] | select(.name == "DB_CONNECT") | .value' pod.yaml

Let’s make it a little harder.

Our new YAML file (users.yaml):

user:
  name: Batur
  surname: Orkun
  gender: male
  active: true
  addresses:
      city: Ankara
      country: Turkey
  subscriptions:
    active:
     - 12345
     - 56789
     - 10267
    passive:
     - 34566
     - 89011

I wonder if the user has any active subscriptions.

yq '.user.subscriptions | has("active")' users.yaml

Output is "true”.

I wonder how many active subscriptions there are.

yq '.user.subscriptions.active | length' users.yaml

Output is “3”

Suppose you don’t know how to use “length”. But it’s a good thing you know Linux.

yq '.user.subscriptions.active' users.yaml | wc -l

I wonder which passive subscriptions have started with “3”

yq  '.user.subscriptions.passive[] | select(. == "3*")' users.yaml

Output is “34566”.

I want to continue with Kubernetes. If you are a DevOps who is working on Kubernetes, you probably already know “yq” or will love it now.

Pretty-printing is possible with yq in CLI.

kubectl get ns kube-system -o yaml | yq '.' -

Deleting fields is possible with yq in CLI. The command below will delete all “annotations” from YAML.

kubectl get ns kube-system -o yaml | yq 'del(.metadata.annotations)'

Or adding new fields. Add to the new annotation:

kubectl get ns kube-system -o yaml | yq '.metadata.annotations.mydate |= "2023-10-31"'

If you want to change the Kubernetes configuration with a new YAML file, you can do it like this:

kubectl get ns batur -o yaml | yq '.metadata.annotations.mydate |= "2023-10-31"' | kubectl apply -f -

You most likely ran into the terminating state when trying to remove a namespace. You tried the parameters “ — force — grace-period=0” but nothing.

kubectl get ns batur -o yaml  | yq '.spec.finalizers = []' | kubectl replace -f -

That is our new YAML file:

services:
  service-1:
    image: image1

  service-2:
    image: image2
    ports:
      - "9090:8080"
  service-4:
    image: image3
    ports:
      - "80:8081"

I wonder what my services are.

yq e '.services | keys'  services.yaml

Output:

- service-1
- service-2
- service-3

Maybe you need multiple pieces of information.

yq e '"Total Services: " + (.services | length), (.services | keys)[]' services.yaml

Total Services: 3
service-1
service-2
service-3

I only need image names that have ports.

yq eval '.services | with_entries(select(.value.ports != null)) | .[] | .image' services.yaml

Output:

image2
image3

There are many advanced “yq” uses. I had to write a book about “yq”, if I wanted to mention it all. You can glance at the link below.

yq