Buildkit on Windows

Anthony Nandaa, 2023/24

TODO

This is a draft area for my todo's, not all of them.

• Set up a build pipeline for e2e testing for WCOW buildkit (on AzDO)
• Craft the current issues identified into specific issue on moby/buildkit and link them. Add help needed tag where possible.
• Fuzz a few dockerfiles for testing, using GPT. (Later on, can see if this can be part of our e2e workflow).
• Revisit to close issue buildkit#616

Integration Tests

Cheatsheet

• Busybox for Windows: registry.k8s.io/e2e-test-images/busybox:1.29-2
• Sample test:
  ​​​​go test --test.run=TestIntegration/TestImageManifestRegistryCacheImportExport/worker=container
  

• Debugging with Delve
  ​​​​dlv test -- --test.run=TestIntegration/TestImageManifestRegistryCacheImportExport/worker=container
  

/client

• client/client_test.go testExportedImageLabels
  • patch: test-export-image-labels.patch
  • run log: https://paste.opensuse.org/pastes/9af7210bd8b5
• testCacheExportCacheKeyLoop:
  • run log: client-272.txt
  • code changes, alpine -> busybox

To be continued

Issues that need investigation

This is a working area for the issues before opening them on moby/buildkit.
Or for collaborative investigation.

Access Issues for ContainerUser in Root (WS2019)

PS C:\Users\Administrator\sample_dockerfile> Set-Content Dockerfile @"

​​​​FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
​​​​RUN mkdir hello
​​​​COPY hello.txt C:/hello/hello.txt
​​​​RUN echo "Goodbye!" >> C:/hello/hello.txt
​​​​CMD ["cmd", "/C", "type C:\\hello\\hello.txt"]

"@
PS C:\Users\Administrator\sample_dockerfile> docker build –no-cache -t hello-buildkit:docker .
Sending build context to Docker daemon 3.072kB
Step 1/5 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
–-> 6ad91fb31728
Step 2/5 : RUN mkdir hello
–-> Running in 83274caa0797
Removing intermediate container 83274caa0797
–-> 1af3e89e5d1e
Step 3/5 : COPY hello.txt C:/hello/hello.txt
–-> 24baa07e955e
Step 4/5 : RUN echo "Goodbye!" >> C:/hello/hello.txt
–-> Running in cac8679cf3d6
Access is denied.
The command 'cmd /S /C echo "Goodbye!" >> C:/hello/hello.txt' returned a non-zero code: 1

TODO: try to run the same command with docker un-elevated to prove ACL enforcement.

Slashes in COPY stanza → #9646

Verify if it's really a regression since docker build throws an error too, until you use front-slashes /:

Step 3/5 : COPY hello.txt C:\hello\hello.txt
failed to copy files: failed to copy file: mkdir \\?\Volume{099f2dd8-e06c-4d8b-ae43-501351a34f93}\C:.: The directory name is invalid.

CMD stanza ignored?

Cross-check why CMD is not run:

PS C:\dev\play\dockerfiles\hello_world2> buildctl build `
>> --frontend=dockerfile.v0 `
>> --local context=. \ `
>> --local dockerfile=. `
>> --output type=image,name=docker.io/0x/hello-buildkit,push=false
[+] Building 8.3s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                 0.0s
 => => transferring dockerfile: 183B                                                                                 0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                       1.8s
 => [internal] load .dockerignore                                                                                    0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [internal] load build context                                                                                    0.1s
 => => transferring context: 133B                                                                                    0.0s
 => CACHED [1/3] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a50  0.1s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06a  0.1s
 => [2/3] COPY hello.txt C:/                                                                                         0.3s
 => [3/3] RUN echo "Goodbye!" >> hello.txt                                                                           2.2s
 => exporting to image                                                                                               3.5s
 => => exporting layers                                                                                              3.4s
 => => exporting manifest sha256:e254603715df63e0d32a1328ee08da269c403717774a298189cc9c247b4f2153                    0.0s
 => => exporting config sha256:2e98fcffba00eace82265deddcc90a8302cfaa90f504bc97239678b04cd5495c                      0.0s
 => => naming to docker.io/0x/hello-buildkit                                                                         0.0s
PS C:\dev\play\dockerfiles\hello_world2> docker build -t hello-bk-repro .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> 0db1879370e5
Step 2/4 : COPY hello.txt C:/
 ---> 3e4e63e377c9
Step 3/4 : RUN echo "Goodbye!" >> hello.txt
 ---> Running in 2da1134ae39f
Removing intermediate container 2da1134ae39f
 ---> 95ed8d34843b
Step 4/4 : CMD ["cmd", "/C", "type C:\\hello.txt"]
 ---> Running in 47c679c37da7
Removing intermediate container 47c679c37da7
 ---> 1e4d61ff9ef1
Successfully built 1e4d61ff9ef1
Successfully tagged hello-bk-repro:latest

Displaying of \ in the progress texts (it’s stripped off)

TBD

Access Denied issue for items at the root

Repro:

• Follow the current example at docs/windows.md
• Using WS2022

You end up with:

PS C:\Users\Administrator\sample_dockerfile> buildctl build `
>> --frontend=dockerfile.v0 `
>> --local context=. \ `
>> --local dockerfile=. `
>> --output type=image,name=docker.io/<your_username>/hello-buildkit,push=true
[+] Building 16.0s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                    0.1s
 => => transferring dockerfile: 183B                                                                                                    0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                                          0.5s
 => [internal] load .dockerignore                                                                                                       0.0s
 => => transferring context: 2B                                                                                                         0.0s
 => [internal] load build context                                                                                                       0.2s
 => => transferring context: 133B                                                                                                       0.0s
 => [1/3] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7   12.4s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7    0.1s
 => => sha256:0de484043f1248d5382233de81d1687426c4c64b6507d0f768eff763724f1bac 116.97MB / 116.97MB                                      2.5s
 => => extracting sha256:0de484043f1248d5382233de81d1687426c4c64b6507d0f768eff763724f1bac                                               9.7s
 => [2/3] COPY hello.txt C:/                                                                                                            0.3s
 => ERROR [3/3] RUN echo "Goodbye!" >> hello.txt                                                                                        2.4s
------
 > [3/3] RUN echo "Goodbye!" >> hello.txt:
1.949 Access is denied.
------
Dockerfile:3
--------------------
   1 |     FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
   2 |     COPY hello.txt C:/
   3 | >>> RUN echo "Goodbye!" >> hello.txt
   4 |     CMD ["cmd", "/C", "type C:\\hello.txt"]
   5 |
--------------------
error: failed to solve: process "cmd /S /C echo \"Goodbye!\" >> hello.txt" did not complete successfully: exit code: 1

This is not a regression:

docker build also handles it in the same way:

PS > docker build -t hello-buildkit:docker .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
ltsc2022: Pulling from windows/nanoserver
0de484043f12: Pull complete
Digest: sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7
Status: Downloaded newer image for mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> 6ad91fb31728
Step 2/4 : COPY hello.txt C:/
 ---> 652f89575cdf
Step 3/4 : RUN echo "Goodbye!" >> hello.txt
 ---> Running in 0540aad493a7
Access is denied.
The command 'cmd /S /C echo "Goodbye!" >> hello.txt' returned a non-zero code: 1

Fix:

Option 1:

• Add USER ContainerAdministrator to the dockerfile, since the default user is a low-priv ContainerUser:
  ​​​​Set-Content Dockerfile @"
  ​​​​FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
  ​​​​USER ContainerAdministrator
  ​​​​COPY hello.txt C:/
  ​​​​RUN echo "Goodbye!" >> hello.txt
  ​​​​CMD ["cmd", "/C", "type C:\\hello.txt"]
  ​​​​"@
  

Option 2:

• If you would like to maintain the low-priv, don't write to the root directory, butter anything inner (and not a system directory like /windows, etc). Therefore, we will modify the dockerfile to:

  ​​​​Set-Content Dockerfile @"
  ​​​​FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
  ​​​​RUN mkdir hello
  ​​​​COPY hello.txt C:/hello/hello.txt
  ​​​​RUN echo "Goodbye!" >> hello\hello.txt
  ​​​​CMD ["cmd", "/C", "type C:\\hello\\hello.txt"]
  ​​​​"@