Lately I have been creating and debugging container images without having the possibility to check the corresponding Dockerfiles. What I will show you today is how we can modify the behaviour of an OCI Image and swap blobs with others.

We will start by modifying a Docker Image similar to the one that I have shown here: we will change the entrypoint and then gain confidence for the next activity; in fact we will then create an OCI Image and change a blob.

NB: we will do the following tasks “the hard way” without relying to other direct methodologies/already working tools to achieve the desired results. The desired result can also be achieved in different (hard) ways

#1 Changing the Manifest config

Our image is this docker.io/jpolidor/chaos-monkey:0.0.1-SNAPSHOT and there is an error in the entrypoint section of the Dockerfile: it won’t run if you don’t specifiy a /config volume. We will try to override this behaviour by specifying that volume as optional.

Let’s pull the image:

> podman run -it  docker.io/jpolidor/chaos-monkey:0.0.1-SNAPSHOT

Trying to pull docker.io/jpolidor/chaos-monkey:0.0.1-SNAPSHOT...
Getting image source signatures
Copying blob sha256:52493f9aff6923b98b73ed7b46325e938ff719f7edde34dd2973d0327ffc27d6
Copying blob sha256:fda4ba87f6fbeebb651625814b981d4100a98c224fef80d562fb33853500f40e
Copying blob sha256:a1f1879bb7de17d50f521ac7c19f3ed9779ded79f26461afb92124ddb1ee7e27
Copying blob sha256:8a4811140f89555dd59ae3c2595ee312b4300407165af869aa82e3edc863e80e
Copying blob sha256:407f249037ea9c2ff1e6b62ac8970ee412a040e64e8a474adaec811b96d1bb08
Copying blob sha256:8fdb1fc20e240e9cae976518305db9f9486caa155fd5fc53e7b3a3285fe8a990
Copying blob sha256:91f7dc022a61cba225bc628b45533035d33e21898a32c4ed398ae03bc8050fd3
Copying blob sha256:4bfc769b577fb861fda86a8451f2aaa00784c0680b2919c35bf680a11dd351b2
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:0e6321645db90f077cfe0bc5e053ce790de8642e5e088e74e7e44a781b1e28d7
Copying config sha256:1b5f81cbce1c7fb94bbc2a3d77d36bb0375a9771206f7f9a90eb1873ac411692
Writing manifest to image destination
Storing signatures
1b5f81cbce1c7fb94bbc2a3d77d36bb0375a9771206f7f9a90eb1873ac41169

Once we have the image locally run it:

***************************
APPLICATION FAILED TO START
***************************

Description:

Config data location '/config/' does not exist

Action:

Check that the value '/config/' is correct, or prefix it with 'optional:'

As you can see there is the error. Let’s check the manifest config. We can do this in two way:

via the ORAS Cli -> oras manifest fetch-config docker.io/jpolidor/chaos-monkey:0.0.1-SNAPSHOT --pretty
by inspecting the blobs

Let’s inspect the blobs but first we need to export the image as tar and explode it.

> podman save --output chaos-monkey.tar --format=oci-archive docker.io/jpolidor/chaos-monkey:0.0.1-SNAPSHOT
> tar -xzvf chaos-monkey.tar
x blobs/
x blobs/sha256/
x blobs/sha256/42b696605cc2c11351e23e04e24e48a01d64f3c90b85bce618c412a7595b95d2
x blobs/sha256/435d77bb73d5c9ec43586d2b5c4cb6f6b83bfe3795855bfc923c4eeb8df02eed
x blobs/sha256/49651fb9e2fc7f8e68929b531ef149dd4985fa77358b619769e6edb8f6cea5e9
x blobs/sha256/5b1b833d0c40ac703d8539f1292160d359adb03e911e418dbb479f599955fe6f
x blobs/sha256/653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3
x blobs/sha256/686c98b07801dcfce953b51e3b028d1fc6d1ab72d130034e91ab24fab1ecf946
x blobs/sha256/6d5664145de1178523f5eb449170c24d0413b518b172611a6969d7c89143a7ec
x blobs/sha256/8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346
x blobs/sha256/a47538ab31a4ff1c3ddf84f8e3ecc4c28e338abae9cf4f57a70285b7ee4cd4b9
x blobs/sha256/bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d
x blobs/sha256/d2388295da93a5b56e8e79798d7bd9cd2d77dece398041360c2876d2d99f14fb
x blobs/sha256/e14c0298e997ecab3e718e7c6f8c4d5f2cc0507a19a666923114ac09422140ce
x index.json
x oci-layout

> rm chaos-monkey.tar

as you can see we have the following files/folder:

oci-layout: it contains the version of the OCI Image layout. In this case 1.0.0
blobs/sha256 folder: it contains all the blobs from of the OCI Image
index.json: is the Image Index file; it points to a specific Image Manifest (usually it’s used to list all the images for different platforms).

Let’s check the index.json, it will guide us to the blob where we will find the Image Manifest (and from there the manifest-config):

> cat index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346",
      "size": 1814,
      "annotations": {
        "org.opencontainers.image.ref.name": "docker.io/jpolidor/chaos-monkey:0.0.1-SNAPSHOT"
      }
    }
  ]
}

ok so we know that the blob is 8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346 from the blob folder.

> cat blobs/sha256/8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3",
    "size": 2711
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:435d77bb73d5c9ec43586d2b5c4cb6f6b83bfe3795855bfc923c4eeb8df02eed",
      "size": 819084
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:686c98b07801dcfce953b51e3b028d1fc6d1ab72d130034e91ab24fab1ecf946",
      "size": 8289593
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5b1b833d0c40ac703d8539f1292160d359adb03e911e418dbb479f599955fe6f",
      "size": 848324
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d2388295da93a5b56e8e79798d7bd9cd2d77dece398041360c2876d2d99f14fb",
      "size": 9512027
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:6d5664145de1178523f5eb449170c24d0413b518b172611a6969d7c89143a7ec",
      "size": 69597944
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:49651fb9e2fc7f8e68929b531ef149dd4985fa77358b619769e6edb8f6cea5e9",
      "size": 113
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:42b696605cc2c11351e23e04e24e48a01d64f3c90b85bce618c412a7595b95d2",
      "size": 39332864
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e14c0298e997ecab3e718e7c6f8c4d5f2cc0507a19a666923114ac09422140ce",
      "size": 88737
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d",
      "size": 34
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a47538ab31a4ff1c3ddf84f8e3ecc4c28e338abae9cf4f57a70285b7ee4cd4b9",
      "size": 5321
    }
  ]
}

Here we go, the manifest for our Image. We’re interested in the Config descriptor

[...]
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3",
    "size": 2711
  },
[...]

it says that the config we are looking for is 653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3.

> cat blobs/sha256/653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3 | jq
{
  "created": "2022-11-10T19:32:27.865963Z",
  "author": "Bazel",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "User": "nonroot",
    "Env": [
      "JAVA_VERSION=17.0.4",
      "LANG=C.UTF-8",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
    ],
    "Entrypoint": [
      "java",
      "-Dfile.encoding=UTF-8",
      "-Dspring.config.additional-location=/config/",
      "org.springframework.boot.loader.JarLauncher"
    ],
    "WorkingDir": "/application"
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:9fce6bd02a21068901a96271d3160c2ce9da9c83c152eeb22312f65226da108c",
      "sha256:00c562532b203116c721c11aee6fe1fe50b5e2068cee2c343c1df1050a866b3e",
      "sha256:e83d4114481dc897d49d5a9a8b68bf71c4f08f1a5c1adff44603c56b958fed53",
      "sha256:45f84957a62ddbeee272fc25653d35160f934dd8bb930bc0d30b80d1cfb7cfd3",
      "sha256:468a6b28e336b9ead462b8128b8d8c60e458a1facc41b5e925fca21b6718c368",
      "sha256:d711cd244d2e61ab7be1d33773e6200de72ced1911a5dc104edffcda4eba845f",
      "sha256:20e75fa20dbab366c9bba8a489fa696cc93bcf7bdbff52c1eba25d4a36cb7149",
      "sha256:750121044cc0f1e14c691e660cb44c29a3b6c85fa8c77e475d7efef2017e7817",
      "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
      "sha256:4e03ec8ca9569c8f2680f127f6ce4be8074923a507cbeacc9c3b3fe965481531"
    ]
  },
  "history": [
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "2022-11-10T17:33:01.754690052Z",
      "created_by": "WORKDIR /application",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2022-11-10T17:33:25.960967208Z",
      "created_by": "COPY /output/dependencies/ ./ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2022-11-10T17:33:25.989440598Z",
      "created_by": "COPY /output/spring-boot-loader/ ./ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2022-11-10T17:33:26.007907811Z",
      "created_by": "COPY /output/snapshot-dependencies/ ./ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2022-11-10T19:32:27.865963Z",
      "created_by": "COPY /output/application/ ./ # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2022-11-10T19:32:27.865963Z",
      "created_by": "USER nonroot",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2022-11-10T19:32:27.865963Z",
      "created_by": "ENTRYPOINT [\"java\" \"-Dfile.encoding=UTF-8\" \"-Dspring.config.additional-location=/config/\" \"org.springframework.boot.loader.JarLauncher\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    }
  ]
}

Here we go! The configuration object for the container. It displays the the environment variables, entrypoint, list of blobs of the image and the history for each layer (blob).

As I said we’re interested to the entrypoint section. We want to change it from this:

[...]
    "Entrypoint": [
      "java",
      "-Dfile.encoding=UTF-8",
      "-Dspring.config.additional-location=/config/",
      "org.springframework.boot.loader.JarLauncher"
    ],
[...]

[...]
    "Entrypoint": [
      "java",
      "-Dfile.encoding=UTF-8",
      "-Dspring.config.additional-location=optional:/config/",
      "org.springframework.boot.loader.JarLauncher"
    ],
[...]

Can we just change it in this way? Well no. Let’s draw a diagram in order to understand the relationship between the Index, Image Manifest and Image-Config Manifest.

As you can see the index.json is what represents the TAG for an Image and references an Image Manifest with that specific sha256; then the Image Manifest references the blobs and the Image config descriptor (NB: both image manifest and config image manifest are blobs).

Then if we change someting we will break the relationships above since the sha256 will change. Also, you have may noticed that there is also a size field: it will change as well. In order to make the change we want to do, we will need to re-compute the size and the sha256 of the layers we will change.

Let’s start.

First of all let’s change the entrypoint as we said above and once done, compute the new sha256, rename the blob and compute the new size:

> openssl sha256 blobs/sha256/653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3
SHA256(blobs/sha256/653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3)= c55fe6619909ebf7032791e7c3fa8ffcd0e5949dd6935cd07fe05aad6528a4fc

> mv blobs/sha256/653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3 blobs/sha256/c55fe6619909ebf7032791e7c3fa8ffcd0e5949dd6935cd07fe05aad6528a4fc

> stat -f%z blobs/sha256/c55fe6619909ebf7032791e7c3fa8ffcd0e5949dd6935cd07fe05aad6528a4fc
2720

So now we need to go back where this image-config was referenced from and change the sha256 and the size (the sha256 of the Image manifest was 8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346):

from:

[...]
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:653c9105f546b2717d9fa62a1215cd344c0fc34a7aade35737d28a86539912b3",
    "size": 2711
  },
[...]

to:

[...]
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:c55fe6619909ebf7032791e7c3fa8ffcd0e5949dd6935cd07fe05aad6528a4fc",
    "size": 2720
  },
[...]

But then we need to re-compute the sha256 and rename this manifest since we changed the content, so:

> openssl sha256 blobs/sha256/8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346
SHA256(blobs/sha256/8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346)= 0435936d5e42d4113575de7299fcf8b7a769dc2893c019956f711823f4fefda4


> mv blobs/sha256/8549ea9700a68925855cf30f9a4236c6806050761957c4ed56845f433fa3b346 blobs/sha256/0435936d5e42d4113575de7299fcf8b7a769dc2893c019956f711823f4fefda4

The last file we need to modify is the index.json since the Image manifest sha256 changed. So now it should be like this:

> cat index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:0435936d5e42d4113575de7299fcf8b7a769dc2893c019956f711823f4fefda4",
      "size": 1814,
      "annotations": {
        "org.opencontainers.image.ref.name": "localhost/chaos-monkey:0.0.1-SNAPSHOT"
      }
    }
  ]
}

I also changed the annotation org.opencontainers.image.ref.name to reference our local image (not the one from the registry we pulled).

At this point we’re ready! Let’s create a tar out of these files and folders

> tar -czvf chaos-monkey.tar -C chaos-monkey .
a .
a ./oci-layout
a ./blobs
a ./index.json
a ./chaos-monkey.tar
a ./blobs/sha256
a ./blobs/sha256/49651fb9e2fc7f8e68929b531ef149dd4985fa77358b619769e6edb8f6cea5e9
a ./blobs/sha256/5b1b833d0c40ac703d8539f1292160d359adb03e911e418dbb479f599955fe6f
a ./blobs/sha256/c55fe6619909ebf7032791e7c3fa8ffcd0e5949dd6935cd07fe05aad6528a4fc
a ./blobs/sha256/d2388295da93a5b56e8e79798d7bd9cd2d77dece398041360c2876d2d99f14fb
a ./blobs/sha256/e14c0298e997ecab3e718e7c6f8c4d5f2cc0507a19a666923114ac09422140ce
a ./blobs/sha256/42b696605cc2c11351e23e04e24e48a01d64f3c90b85bce618c412a7595b95d2
a ./blobs/sha256/686c98b07801dcfce953b51e3b028d1fc6d1ab72d130034e91ab24fab1ecf946
a ./blobs/sha256/a47538ab31a4ff1c3ddf84f8e3ecc4c28e338abae9cf4f57a70285b7ee4cd4b9
a ./blobs/sha256/435d77bb73d5c9ec43586d2b5c4cb6f6b83bfe3795855bfc923c4eeb8df02eed
a ./blobs/sha256/bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d
a ./blobs/sha256/0435936d5e42d4113575de7299fcf8b7a769dc2893c019956f711823f4fefda4
a ./blobs/sha256/6d5664145de1178523f5eb449170c24d0413b518b172611a6969d7c89143a7ec

NB: note that we’re not creating a root folder in the tar.

Then create the Image:

podman load -i chaos-monkey.tar
Loaded image: localhost/chaos-monkey:0.0.1-SNAPSHOT

And run it

podman run -it localhost/chaos-monkey:0.0.1-SNAPSHOT

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.5)

It worked! (the application won’t start since we’re not specifying the property file and a profile but that’s not the point of this tutorial).

#2 Swap a blob

Let’s start by creating an OCI Image with a Dockerfile:

FROM bash:5.2.15
COPY ./file.txt .
ENTRYPOINT [ "cat", "./file.txt" ]

The txt file contains the text “Hello Friend”. Now build the image, run it, save the image as tar and explode it:

> podman build . -t hello-friend
STEP 1/3: FROM bash:5.2.15
STEP 2/3: COPY ./file.txt .
--> Using cache 4418421f2e7fb5d3eed205e1216a8cd2a405a687aac351387469f7cbc620f983
--> 4418421f2e7
STEP 3/3: ENTRYPOINT [ "cat", "./file.txt" ]
COMMIT hello-friend
--> bdae866ed07
Successfully tagged localhost/hello-friend:latest
bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b
> podman run -it localhost/hello-friend
Hello Friend!
> podman save --output hello-friend.tar --format=oci-archive localhost/hello-friend
> tar -xzvf hello-friend
> rm hello-friend

What we want to achieve now is to change the text from “Hello Friend” to “Hello friend, how are you?”.

Let’s find where is the image manifest:

cat index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c",
      "size": 1070,
      "annotations": {
        "org.opencontainers.image.ref.name": "localhost/hello-friend:latest"
      }
    }
  ]
}

so 9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c

cat blobs/sha256/9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b",
    "size": 5190
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:dc4c970b3f26e8ac79156f3c5f19f19e9098a1249a8a0add369cad0601d30c27",
      "size": 2924467
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d289e0002204f578b55a976a2c169a6ed25b21416dee930339e65d51eeb46eda",
      "size": 2820708
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f60351486597d3001ab3adbcae10e2b96c563e8c4d9dcf114937b54db8208dab",
      "size": 351
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:dd369cc6b9089c982c7ef3e9d35cb6358160c6e148a663fbc6415358f3785269",
      "size": 120
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:b6e8634ffbb847e37fd1530383fd877a88baacfeac17ffbd09207ce0283c0065",
    "org.opencontainers.image.base.name": "docker.io/library/bash:5.2.15"
  }
}

As you can see we have 4 layers. We need to understand which of these layers is the one that contains our txt file. We understand better what is going on by checking the Image config manifest:

cat blobs/sha256/bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b | jq
{
  "created": "2023-02-19T14:08:08.718234205Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "_BASH_VERSION=5.2.15",
      "_BASH_BASELINE=5.2.15",
      "_BASH_BASELINE_PATCH=15",
      "_BASH_LATEST_PATCH=15"
    ],
    "Entrypoint": [
      "cat",
      "./file.txt"
    ],
    "Labels": {
      "io.buildah.version": "1.28.0"
    }
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:aa5968d388b8652cd305e0e037751228967839d83d0cafbde5debf0b092e7c42",
      "sha256:ec41f5ffa8cc86489d1ca8a79bf5a95a41d0c2c79a745fc5d25ac09bff429c3d",
      "sha256:1b077c887b85d5ed83f6a2bff381201c5e2dac2549e3e53cbece12bd9f832cd1",
      "sha256:e131a00fba7bd93414f52489ba50c936322289f9237b282c0882e9c637716a97",
    ]
  },
  "history": [
    {
      "created": "2023-02-11T04:46:50.143309571Z",
      "created_by": "/bin/sh -c #(nop) ADD file:ac5fb7eb0d68040d948989f0a50914d0d4a6b631cfe76b508eecd82eb7d46953 in / "
    },
    {
      "created": "2023-02-11T04:46:50.26254182Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
      "empty_layer": true
    },
    {
      "created": "2023-02-11T07:19:05.522790472Z",
      "created_by": "/bin/sh -c #(nop)  ENV _BASH_VERSION=5.2.15",
      "empty_layer": true
    },
    {
      "created": "2023-02-11T07:19:05.623956846Z",
      "created_by": "/bin/sh -c #(nop)  ENV _BASH_BASELINE=5.2.15",
      "empty_layer": true
    },
    {
      "created": "2023-02-11T07:19:05.720704216Z",
      "created_by": "/bin/sh -c #(nop)  ENV _BASH_BASELINE_PATCH=15",
      "empty_layer": true
    },
    {
      "created": "2023-02-11T07:19:05.822575316Z",
      "created_by": "/bin/sh -c #(nop)  ENV _BASH_LATEST_PATCH=15",
      "empty_layer": true
    },
    {
      "created": "2023-02-11T07:19:46.414745975Z",
      "created_by": "/bin/sh -c set -eux; \t\tapk add --no-cache --virtual .build-deps \t\tbison \t\tcoreutils \t\tdpkg-dev dpkg \t\tgcc \t\tlibc-dev \t\tmake \t\tncurses-dev \t\ttar \t; \t\twget -O bash.tar.gz \"https://ftp.gnu.org/gnu/bash/bash-$_BASH_BASELINE.tar.gz\"; \twget -O bash.tar.gz.sig \"https://ftp.gnu.org/gnu/bash/bash-$_BASH_BASELINE.tar.gz.sig\"; \t\t: \"${_BASH_BASELINE_PATCH:=0}\" \"${_BASH_LATEST_PATCH:=0}\"; \tif [ \"$_BASH_LATEST_PATCH\" -gt \"$_BASH_BASELINE_PATCH\" ]; then \t\tmkdir -p bash-patches; \t\tfirst=\"$(printf '%03d' \"$(( _BASH_BASELINE_PATCH + 1 ))\")\"; \t\tlast=\"$(printf '%03d' \"$_BASH_LATEST_PATCH\")\"; \t\tmajorMinor=\"${_BASH_VERSION%.*}\"; \t\tfor patch in $(seq -w \"$first\" \"$last\"); do \t\t\turl=\"https://ftp.gnu.org/gnu/bash/bash-$majorMinor-patches/bash${majorMinor//./}-$patch\"; \t\t\twget -O \"bash-patches/$patch\" \"$url\"; \t\t\twget -O \"bash-patches/$patch.sig\" \"$url.sig\"; \t\tdone; \tfi; \t\tapk add --no-cache --virtual .gpg-deps gnupg; \texport GNUPGHOME=\"$(mktemp -d)\"; \tgpg --batch --keyserver keyserver.ubuntu.com --recv-keys 7C0135FB088AAF6C66C650B9BB5869F064EA74AB; \tgpg --batch --verify bash.tar.gz.sig bash.tar.gz; \trm bash.tar.gz.sig; \tif [ -d bash-patches ]; then \t\tfor sig in bash-patches/*.sig; do \t\t\tp=\"${sig%.sig}\"; \t\t\tgpg --batch --verify \"$sig\" \"$p\"; \t\t\trm \"$sig\"; \t\tdone; \tfi; \tgpgconf --kill all; \trm -rf \"$GNUPGHOME\"; \tapk del --no-network .gpg-deps; \t\tmkdir -p /usr/src/bash; \ttar \t\t--extract \t\t--file=bash.tar.gz \t\t--strip-components=1 \t\t--directory=/usr/src/bash \t; \trm bash.tar.gz; \t\tif [ -d bash-patches ]; then \t\tapk add --no-cache --virtual .patch-deps patch; \t\tfor p in bash-patches/*; do \t\t\tpatch \t\t\t\t--directory=/usr/src/bash \t\t\t\t--input=\"$(readlink -f \"$p\")\" \t\t\t\t--strip=0 \t\t\t; \t\t\trm \"$p\"; \t\tdone; \t\trmdir bash-patches; \t\tapk del --no-network .patch-deps; \tfi; \t\tcd /usr/src/bash; \tgnuArch=\"$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)\"; \t./configure \t\t--build=\"$gnuArch\" \t\t--enable-readline \t\t--with-curses \t\t--without-bash-malloc \t|| { \t\tcat >&2 config.log; \t\tfalse; \t}; \tmake -j \"$(nproc)\"; \tmake install; \tcd /; \trm -r /usr/src/bash; \t\trm -rf \t\t/usr/local/share/doc/bash/*.html \t\t/usr/local/share/info \t\t/usr/local/share/locale \t\t/usr/local/share/man \t; \t\trunDeps=\"$( \t\tscanelf --needed --nobanner --format '%n#p' --recursive /usr/local \t\t\t| tr ',' '\\n' \t\t\t| sort -u \t\t\t| awk 'system(\"[ -e /usr/local/lib/\" $1 \" ]\") == 0 { next } { print \"so:\" $1 }' \t)\"; \tapk add --no-network --virtual .bash-rundeps $runDeps; \tapk del --no-network .build-deps; \t\t[ \"$(which bash)\" = '/usr/local/bin/bash' ]; \tbash --version; \t[ \"$(bash -c 'echo \"${BASH_VERSION%%[^0-9.]*}\"')\" = \"$_BASH_VERSION\" ]; \tbash -c 'help' > /dev/null"
    },
    {
      "created": "2023-02-11T07:19:46.551038638Z",
      "created_by": "/bin/sh -c #(nop) COPY file:651b3bebeba8be9162c56b3eb561199905235f3e1c7811232b6c9f48ac333651 in /usr/local/bin/ "
    },
    {
      "created": "2023-02-11T07:19:46.644105509Z",
      "created_by": "/bin/sh -c #(nop)  ENTRYPOINT [\"docker-entrypoint.sh\"]",
      "empty_layer": true
    },
    {
      "created": "2023-02-11T07:19:46.739145156Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
      "empty_layer": true
    },
    {
      "created": "2023-02-19T14:07:33.006303467Z",
      "created_by": "/bin/sh -c #(nop) COPY file:e7baf5ea910940be73aba2e7f2e6abcff5c9c7c8bf577b4dd852c644fff0129f in . ",
      "comment": "FROM docker.io/library/bash:5.2.15"
    },
    {
      "created": "2023-02-19T14:08:08.718872297Z",
      "created_by": "/bin/sh -c #(nop) ENTRYPOINT [ \"cat\", \"./file.txt\" ]",
      "comment": "FROM 4418421f2e7f",
      "empty_layer": true
    }
  ]
}

As you can see, from here we can check the history and now we know that the COPY command is the fourth layer, with sha256 equal to dd369cc6b9089c982c7ef3e9d35cb6358160c6e148a663fbc6415358f3785269 (NB: Not each command in the dockerfile produces a layer. In fact only: FROM, COPY/ADD, RUN produce layers).

Let’s check:

tar -zxvf blobs/sha256/dd369cc6b9089c982c7ef3e9d35cb6358160c6e148a663fbc6415358f3785269
x file.txt
> cat file.txt
Hello Friend!

Ok so now create our txt file with the new content, tar it and then compute the sha256.:

> echo "Hello friend, How are you?" > file.txt
> cat file.txt
Hello friend, How are you?


> tar -cvf file.tar file.txt
a file.txt
> openssl sha256 file.tar
SHA256(file.tar)= ee3a990d77d661d53114c77fa5914708f27e764bebaf13046595d2ee98fc4e21

Once we have the sha256 of the tar we want to append it to the diff_ids of the image manifest config and delete the previous sha256 in the array list.

So we end up having this image config

cat blobs/sha256/bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b | jq
{
[...]
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:aa5968d388b8652cd305e0e037751228967839d83d0cafbde5debf0b092e7c42",
      "sha256:ec41f5ffa8cc86489d1ca8a79bf5a95a41d0c2c79a745fc5d25ac09bff429c3d",
      "sha256:1b077c887b85d5ed83f6a2bff381201c5e2dac2549e3e53cbece12bd9f832cd1",
      "sha256:ee3a990d77d661d53114c77fa5914708f27e764bebaf13046595d2ee98fc4e21"
    ]
  },
[...]

Since we modified it we need to recompute the sha256 of the image config file and change the name accordingly:

> openssl sha256 bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b
SHA256(bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b)= 8e7eeb8ef7f1174a9a28949956ec73405e2675a0a6f6eb6997069c354f3d5925

Now we need to compress the tar with gzip, compute the size and move it inside the blobs folder

> openssl sha256 file.tar.gz
SHA256(file.tar.gz)= 386c4318de69a3ffd484fdc666dacac7c16744e5b5212aa2508bd82188e4411e
> stat -f%z 386c4318de69a3ffd484fdc666dacac7c16744e5b5212aa2508bd82188e4411e
231
> mv file.tar.gz blobs/sha256/386c4318de69a3ffd484fdc666dacac7c16744e5b5212aa2508bd82188e4411e

Delete the old blob that contained the old file.txt

> rm blobs/sha256/dd369cc6b9089c982c7ef3e9d35cb6358160c6e148a663fbc6415358f3785269

Now we just need to change the sha256 references and bytes size in our image manifest, recompute the sha256 of the image manifest and change the image index to reference the new sha256.

So, from this image manifest

cat blobs/sha256/9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:bdae866ed0739c752ee71fdfd76b3b169c6b53e264f5a8b1e3a3accacd77c40b",
    "size": 5190
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:dc4c970b3f26e8ac79156f3c5f19f19e9098a1249a8a0add369cad0601d30c27",
      "size": 2924467
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d289e0002204f578b55a976a2c169a6ed25b21416dee930339e65d51eeb46eda",
      "size": 2820708
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f60351486597d3001ab3adbcae10e2b96c563e8c4d9dcf114937b54db8208dab",
      "size": 351
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:dd369cc6b9089c982c7ef3e9d35cb6358160c6e148a663fbc6415358f3785269",
      "size": 120
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:b6e8634ffbb847e37fd1530383fd877a88baacfeac17ffbd09207ce0283c0065",
    "org.opencontainers.image.base.name": "docker.io/library/bash:5.2.15"
  }
}

change the last layer in order to reflect the new file sha256 and size:

[...]
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:386c4318de69a3ffd484fdc666dacac7c16744e5b5212aa2508bd82188e4411e",
      "size": 231
    }
[...]

and also the config descriptor digest:

  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:8e7eeb8ef7f1174a9a28949956ec73405e2675a0a6f6eb6997069c354f3d5925",
    "size": 5190
  },

Now Get the new sha256 of this file and overwrite the name

> openssl sha256 blobs/sha256/9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c
SHA256(blobs/sha256/9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c)= 5c71cd63836cf0ff13a3994891668e06d666ce36db1ed686b68e4507a406857b

> mv blobs/sha256/9778b85a650506475541bfb98829c60a4fbe77beacbd9b4beabc7c33e20c290c blobs/sha256/5c71cd63836cf0ff13a3994891668e06d666ce36db1ed686b68e4507a406857b

Then change the reference from the index.json to reflect the change above

cat index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:5c71cd63836cf0ff13a3994891668e06d666ce36db1ed686b68e4507a406857b",
      "size": 1070,
      "annotations": {
        "org.opencontainers.image.ref.name": "localhost/hello-friend:latest"
      }
    }
  ]
}

We’re ready to create the tar and load it with podman!

> tar -czvf hello-friend.tar -C hello-friend .
a .
a ./oci-layout
a ./blobs
a ./index.json
a ./blobs/sha256
a ./blobs/sha256/d289e0002204f578b55a976a2c169a6ed25b21416dee930339e65d51eeb46eda
a ./blobs/sha256/5c71cd63836cf0ff13a3994891668e06d666ce36db1ed686b68e4507a406857b
a ./blobs/sha256/dc4c970b3f26e8ac79156f3c5f19f19e9098a1249a8a0add369cad0601d30c27
a ./blobs/sha256/386c4318de69a3ffd484fdc666dacac7c16744e5b5212aa2508bd82188e4411e
a ./blobs/sha256/f60351486597d3001ab3adbcae10e2b96c563e8c4d9dcf114937b54db8208dab
a ./blobs/sha256/8e7eeb8ef7f1174a9a28949956ec73405e2675a0a6f6eb6997069c354f3d5925

> podman load -i hello-friend.tar
Loaded image: localhost/hello-friend:latest

> podman run -it localhost/hello-friend:latest
Hello friend, How are you?

We made it. Doing these changes manually is a very error-prone process (I admit I needed to redo all the commands above multiple times :D ) so it is always better to design a diagram as I did for the first task in order to keep track of the digests and references.

#1 Changing the Manifest config#

#2 Swap a blob#

#1 Changing the Manifest config

#2 Swap a blob