# Buildkit on Windows
Anthony Nandaa, 2023/24
## TODO
> This is a draft area for my todo's, not all of them.
- [ ] Set up a build pipeline for e2e testing for WCOW buildkit (on AzDO)
- [ ] Craft the current issues identified into specific issue on `moby/buildkit` and link them. Add `help needed` tag where possible.
- [ ] Fuzz a few dockerfiles for testing, using GPT. (Later on, can see if this can be part of our e2e workflow).
- [ ] Revisit to close issue [`buildkit#616`](https://github.com/moby/buildkit/issues/616)
## Integration Tests
## Cheatsheet
- Busybox for Windows: `registry.k8s.io/e2e-test-images/busybox:1.29-2`
- Sample test:
```bash
go test --test.run=TestIntegration/TestImageManifestRegistryCacheImportExport/worker=container
```
- Debugging with Delve
```bash
dlv test -- --test.run=TestIntegration/TestImageManifestRegistryCacheImportExport/worker=container
```
### `/client`
- `client/client_test.go testExportedImageLabels`
- patch: [test-export-image-labels.patch](https://github.com/moby/buildkit/files/13725388/test-export-image-labels.patch)
- run log: https://paste.opensuse.org/pastes/9af7210bd8b5
- `testCacheExportCacheKeyLoop`:
- run log: [client-272.txt](https://github.com/moby/buildkit/files/13725685/client-272.txt)
- code changes, alpine -> busybox
-
> To be continued
## Issues that need investigation
> This is a working area for the issues before opening them on `moby/buildkit`.
> Or for collaborative investigation.
### Access Issues for `ContainerUser` in Root (WS2019)
PS C:\Users\Administrator\sample_dockerfile> Set-Content Dockerfile @"
>> FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
>> RUN mkdir hello
>> COPY hello.txt C:/hello/hello.txt
>> RUN echo "Goodbye!" >> C:/hello/hello.txt
>> CMD ["cmd", "/C", "type C:\\hello\\hello.txt"]
>> "@
PS C:\Users\Administrator\sample_dockerfile> docker build --no-cache -t hello-buildkit:docker .
Sending build context to Docker daemon 3.072kB
Step 1/5 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
---> 6ad91fb31728
Step 2/5 : RUN mkdir hello
---> Running in 83274caa0797
Removing intermediate container 83274caa0797
---> 1af3e89e5d1e
Step 3/5 : COPY hello.txt C:/hello/hello.txt
---> 24baa07e955e
Step 4/5 : RUN echo "Goodbye!" >> C:/hello/hello.txt
---> Running in cac8679cf3d6
Access is denied.
The command 'cmd /S /C echo "Goodbye!" >> C:/hello/hello.txt' returned a non-zero code: 1
> TODO: try to run the same command with `docker` un-elevated to prove ACL enforcement.
### Slashes in `COPY` stanza → #9646
Verify if it's really a regression since `docker build` throws an error too, until you use front-slashes `/`:
```
Step 3/5 : COPY hello.txt C:\hello\hello.txt
failed to copy files: failed to copy file: mkdir \\?\Volume{099f2dd8-e06c-4d8b-ae43-501351a34f93}\C:.: The directory name is invalid.
```
### `CMD` stanza ignored?
Cross-check why `CMD` is not run:
```powershell
PS C:\dev\play\dockerfiles\hello_world2> buildctl build `
>> --frontend=dockerfile.v0 `
>> --local context=. \ `
>> --local dockerfile=. `
>> --output type=image,name=docker.io/0x/hello-buildkit,push=false
[+] Building 8.3s (8/8) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 183B 0.0s
=> [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022 1.8s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 133B 0.0s
=> CACHED [1/3] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a50 0.1s
=> => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06a 0.1s
=> [2/3] COPY hello.txt C:/ 0.3s
=> [3/3] RUN echo "Goodbye!" >> hello.txt 2.2s
=> exporting to image 3.5s
=> => exporting layers 3.4s
=> => exporting manifest sha256:e254603715df63e0d32a1328ee08da269c403717774a298189cc9c247b4f2153 0.0s
=> => exporting config sha256:2e98fcffba00eace82265deddcc90a8302cfaa90f504bc97239678b04cd5495c 0.0s
=> => naming to docker.io/0x/hello-buildkit 0.0s
PS C:\dev\play\dockerfiles\hello_world2> docker build -t hello-bk-repro .
Sending build context to Docker daemon 3.072kB
Step 1/4 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
---> 0db1879370e5
Step 2/4 : COPY hello.txt C:/
---> 3e4e63e377c9
Step 3/4 : RUN echo "Goodbye!" >> hello.txt
---> Running in 2da1134ae39f
Removing intermediate container 2da1134ae39f
---> 95ed8d34843b
Step 4/4 : CMD ["cmd", "/C", "type C:\\hello.txt"]
---> Running in 47c679c37da7
Removing intermediate container 47c679c37da7
---> 1e4d61ff9ef1
Successfully built 1e4d61ff9ef1
Successfully tagged hello-bk-repro:latest
```
### Displaying of `\` in the progress texts (it’s stripped off)
TBD
### `Access Denied` issue for items at the root
**Repro:**
- Follow the current example at [docs/windows.md]()
- Using WS2022
You end up with:
```
PS C:\Users\Administrator\sample_dockerfile> buildctl build `
>> --frontend=dockerfile.v0 `
>> --local context=. \ `
>> --local dockerfile=. `
>> --output type=image,name=docker.io/<your_username>/hello-buildkit,push=true
[+] Building 16.0s (7/7) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 183B 0.0s
=> [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.2s
=> => transferring context: 133B 0.0s
=> [1/3] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7 12.4s
=> => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7 0.1s
=> => sha256:0de484043f1248d5382233de81d1687426c4c64b6507d0f768eff763724f1bac 116.97MB / 116.97MB 2.5s
=> => extracting sha256:0de484043f1248d5382233de81d1687426c4c64b6507d0f768eff763724f1bac 9.7s
=> [2/3] COPY hello.txt C:/ 0.3s
=> ERROR [3/3] RUN echo "Goodbye!" >> hello.txt 2.4s
------
> [3/3] RUN echo "Goodbye!" >> hello.txt:
1.949 Access is denied.
------
Dockerfile:3
--------------------
1 | FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
2 | COPY hello.txt C:/
3 | >>> RUN echo "Goodbye!" >> hello.txt
4 | CMD ["cmd", "/C", "type C:\\hello.txt"]
5 |
--------------------
error: failed to solve: process "cmd /S /C echo \"Goodbye!\" >> hello.txt" did not complete successfully: exit code: 1
```
**This is not a regression:**
`docker build` also handles it in the same way:
```
PS > docker build -t hello-buildkit:docker .
Sending build context to Docker daemon 3.072kB
Step 1/4 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
ltsc2022: Pulling from windows/nanoserver
0de484043f12: Pull complete
Digest: sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7
Status: Downloaded newer image for mcr.microsoft.com/windows/nanoserver:ltsc2022
---> 6ad91fb31728
Step 2/4 : COPY hello.txt C:/
---> 652f89575cdf
Step 3/4 : RUN echo "Goodbye!" >> hello.txt
---> Running in 0540aad493a7
Access is denied.
The command 'cmd /S /C echo "Goodbye!" >> hello.txt' returned a non-zero code: 1
```
**Fix:**
**Option 1:**
- Add `USER ContainerAdministrator` to the dockerfile, since the default user is a low-priv `ContainerUser`:
```powershell
Set-Content Dockerfile @"
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator
COPY hello.txt C:/
RUN echo "Goodbye!" >> hello.txt
CMD ["cmd", "/C", "type C:\\hello.txt"]
"@
```
**Option 2:**
- If you would like to maintain the low-priv, don't write to the root directory, butter anything inner (and not a system directory like `/windows`, etc). Therefore, we will modify the `dockerfile` to:
```powershell
Set-Content Dockerfile @"
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
RUN mkdir hello
COPY hello.txt C:/hello/hello.txt
RUN echo "Goodbye!" >> hello\hello.txt
CMD ["cmd", "/C", "type C:\\hello\\hello.txt"]
"@
Sign in with Wallet
Connect another wallet