Simple Dockerfile does not work anymore

[waterkip]

I have a very small project that has a Dockerfile that starts like this:

FROM perl:latest
COPY . .

This does not work anymore due to

#6 [2/3] COPY . .
#6 ERROR: cannot copy to non-directory: /var/lib/docker/overlay2/y3net4vk3shghk747m0mxj34s/merged/lib
------
 > [2/3] COPY . .:
------
Dockerfile:3
--------------------
   1 |     FROM perl:latest
   2 |     #WORKDIR /tmp/foo
   3 | >>> COPY . .
   4 |     RUN cpanm .
   5 |
--------------------
ERROR: failed to solve: cannot copy to non-directory: /var/lib/docker/overlay2/y3net4vk3shghk747m0mxj34s/merged/lib

It doesn't work for 5.36/5.38, but it does work for 5.34

FROM perl:5.34 # works
COPY . .

Do you know if there were any changes on the hub that could cause this? I didn't notice it earlier.
The docker history command shows that the WORKDIR on all (inspected) images is /, which is the default for docker as I understand.

[waterkip]

When I change the default WORKDIR in the image (in generate.pl), and rebuild it and than run the Dockerfile all is well. It seems that WORKDIR / is somehow.. null and void?

[zakame]

Hi @waterkip, thanks for the report! 🙏

This is interesting - I could not reproduce it in a fresh pull of perl:5.38:

Details

 $  mkdir test-copy-workdir                                              
 $  cd test-copy-workdir/        
test-copy-workdir $  cat <<EOF >Dockerfile            
heredoc> FROM perl:5.38  
heredoc> COPY . .
heredoc> EOF                      
test-copy-workdir $  docker build -t test .        
[+] Building 18.8s (7/7) FINISHED                                                                                  docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 61B                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/perl:5.38                                                                       4.1s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 61B                                                                                                   0.0s
 => [1/2] FROM docker.io/library/perl:5.38@sha256:685afd4265ed42ed55ee2ed9906342dcf2a174210b14ed72735e2d33679cd83b                14.2s
 => => resolve docker.io/library/perl:5.38@sha256:685afd4265ed42ed55ee2ed9906342dcf2a174210b14ed72735e2d33679cd83b                 0.0s
 => => sha256:d8ae43322bebbc923a69555f5eb3c463e1c484ce45000a5cd5e946e181782ec8 6.03kB / 6.03kB                                     0.0s
 => => sha256:fc9ce7290e7e014f69623715ba619a0d3488a7b6ecbf09e55e11160365691192 49.59MB / 49.59MB                                   2.6s
 => => sha256:73373011d63f722a4ac6bc23ce7831a17f19e27ef530544642f060aa1d6b028b 23.57MB / 23.57MB                                   7.3s
 => => sha256:685afd4265ed42ed55ee2ed9906342dcf2a174210b14ed72735e2d33679cd83b 988B / 988B                                         0.0s
 => => sha256:47d50b4e2019fed9ce10210740fbf380c590c468bfc313237b37fb407c936c55 1.58kB / 1.58kB                                     0.0s
 => => sha256:87dafffb02e2671e7e5cc07c996fed89f6ccc8e3276ce2dc1bbfb81502cdd795 63.99MB / 63.99MB                                   3.7s
 => => extracting sha256:fc9ce7290e7e014f69623715ba619a0d3488a7b6ecbf09e55e11160365691192                                          1.4s
 => => sha256:f9cd1c60bb2c0a9bd6653d88494ac5f855634c2ec64a14cfa547ab2e257c6519 202.42MB / 202.42MB                                 8.8s
 => => sha256:413acd89fecae00634847cd8ba73940b4aa1ee9358b38f1ead4d2c1922fffe4b 167B / 167B                                         4.3s
 => => sha256:691f6551a46955a8edb39e66eff9b83bcdfa32015bc3e8abf6257f90904ebcae 15.61MB / 15.61MB                                   6.4s
 => => extracting sha256:73373011d63f722a4ac6bc23ce7831a17f19e27ef530544642f060aa1d6b028b                                          0.4s
 => => extracting sha256:87dafffb02e2671e7e5cc07c996fed89f6ccc8e3276ce2dc1bbfb81502cdd795                                          1.6s
 => => extracting sha256:f9cd1c60bb2c0a9bd6653d88494ac5f855634c2ec64a14cfa547ab2e257c6519                                          3.9s
 => => extracting sha256:413acd89fecae00634847cd8ba73940b4aa1ee9358b38f1ead4d2c1922fffe4b                                          0.0s
 => => extracting sha256:691f6551a46955a8edb39e66eff9b83bcdfa32015bc3e8abf6257f90904ebcae                                          0.5s
 => [2/2] COPY . .                                                                                                                 0.4s
 => exporting to image                                                                                                             0.0s
 => => exporting layers                                                                                                            0.0s
 => => writing image sha256:33147b813e050ce23b12b31caa61f7940488b4499c81f09ef151e83195148d28                                       0.0s
 => => naming to docker.io/library/test                                                                                            0.0s

What's Next?
  View summary of image vulnerabilities and recommendations → docker scout quickview
test-copy-workdir $  docker info           
Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /Users/zakame/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /Users/zakame/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/zakame/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/zakame/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/zakame/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/zakame/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/zakame/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.16.1
    Path:     /Users/zakame/.docker/cli-plugins/docker-scout

Server:
 Containers: 35
  Running: 34
  Paused: 0
  Stopped: 1
 Images: 37
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.668GiB
 Name: docker-desktop
 ID: 0e92b02b-079a-4a7e-a37a-0e3dd08762c9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

and seems to work on perl:5.36 and latest as well:

Details

test-copy-workdir $  cat <<EOF >Dockerfile
FROM perl:latest
COPY . .
EOF
test-copy-workdir $  docker build -t test .
[+] Building 2.7s (7/7) FINISHED                                                                                   docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 63B                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/perl:latest                                                                     2.7s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 63B                                                                                                   0.0s
 => CACHED [1/2] FROM docker.io/library/perl:latest@sha256:685afd4265ed42ed55ee2ed9906342dcf2a174210b14ed72735e2d33679cd83b        0.0s
 => [2/2] COPY . .                                                                                                                 0.0s
 => exporting to image                                                                                                             0.0s
 => => exporting layers                                                                                                            0.0s
 => => writing image sha256:618697f021e1d25eddcda6ac48c82486146f7ce6bcd1aabf638e3beea2674901                                       0.0s
 => => naming to docker.io/library/test                                                                                            0.0s

What's Next?
  View summary of image vulnerabilities and recommendations → docker scout quickview
test-copy-workdir $  cat <<EOF >Dockerfile 
FROM perl:5.36  
COPY . .
EOF
test-copy-workdir $  docker build -t test .
[+] Building 4.8s (7/7) FINISHED                                                                                   docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 61B                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/perl:5.36                                                                       2.8s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 61B                                                                                                   0.0s
 => [1/2] FROM docker.io/library/perl:5.36@sha256:576f9fe38bf8afb918154cf63e990e0e5d8918efbe1b81ee4433e25912304384                 1.8s
 => => resolve docker.io/library/perl:5.36@sha256:576f9fe38bf8afb918154cf63e990e0e5d8918efbe1b81ee4433e25912304384                 0.0s
 => => sha256:576f9fe38bf8afb918154cf63e990e0e5d8918efbe1b81ee4433e25912304384 988B / 988B                                         0.0s
 => => sha256:fed9f994e62a8fe55357a431205c6b718730bb2463c19496c606551511f228ca 1.58kB / 1.58kB                                     0.0s
 => => sha256:f4ee4efe74c76d08052c7216666a0574f78ebbbc044297090dcc6fcb71a4ad72 6.03kB / 6.03kB                                     0.0s
 => => sha256:7713a111e1553a1426bb33b1129f5132bf1660c1a1424557e59b6d6469e55494 15.21MB / 15.21MB                                   1.2s
 => => extracting sha256:7713a111e1553a1426bb33b1129f5132bf1660c1a1424557e59b6d6469e55494                                          0.5s
 => [2/2] COPY . .                                                                                                                 0.2s
 => exporting to image                                                                                                             0.0s
 => => exporting layers                                                                                                            0.0s
 => => writing image sha256:9e6f0fba712b999d9b8f3cec7a088979e0fbd354d2ad898fde5353bc77e92716                                       0.0s
 => => naming to docker.io/library/test                                                                                            0.0s

What's Next?
  View summary of image vulnerabilities and recommendations → docker scout quickview

Is the build context in a path where you don't have full ownership (e.g. outside your user's home directory?)

[waterkip]

I have full ownership locally. I noticed it because a gitlab pipeline failed as well.

$ docker info
Client: Docker Engine - Community
 Version:    24.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 34
  Running: 0
  Paused: 0
  Stopped: 34
 Images: 123
 Server Version: 24.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.3.0-2-amd64
 Operating System: Debian GNU/Linux trixie/sid
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.2GiB
 Name: neptune
 ID: GVRD:APM4:JC7K:EUA6:LKOB:NNZU:VYKL:VKSJ:GEVQ:4PP6:4IEH:WI3J
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 22
  Goroutines: 35
  System Time: 2023-08-02T01:00:24.105663555-04:00
  EventsListeners: 0
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false```

[Tekki]

Why don't we set the workdir to something like /usr/src/app? I add this to all my images that are based on this one, just to be sure to not run into such problems.

[waterkip]

One step closer.

The problem is that my repo has a lib dir, when the workdir is set to / you have /lib. It cannot copy that into layer. I did some testing, it works with debian:buster and debian:bullseye, but it does not work with debian:bookworm.

Why don't we set the workdir to something like /usr/src/app? I add this to all my images that are based on this one, just to be sure to not run into such problems.

That was my workaround/solution as well, add a different WORKDIR. I have a patch ready for that. Docker uses / as a default, but I was wondering why it broke now. Seems bookworm is the problem. Most of my projects use a baselayer that has a WORKDIR set to something else, so I haven't noticed the change, but this is a rather barebone project.

[waterkip]

A little bit of testing showed some more information and I think I found what is going on. It is debian bookworm that does things a bit different.

The behaviour isn't present when you do this: COPY lib /lib. But it is present when you do this: COPY . /

What changed is that in bookworm the lib dir has become a symlink to /usr/lib. So now the COPY command fails because it wants to copy a directory into a symlink. This might be a problem with how Docker deals with symlinks?

[zakame]

One step closer.

The problem is that my repo has a lib dir, when the workdir is set to / you have /lib. It cannot copy that into layer. I did some testing, it works with debian:buster and debian:bullseye, but it does not work with debian:bookworm.

Why don't we set the workdir to something like /usr/src/app? I add this to all my images that are based on this one, just to be sure to not run into such problems.

That was my workaround/solution as well, add a different WORKDIR. I have a patch ready for that. Docker uses / as a default, but I was wondering why it broke now. Seems bookworm is the problem. Most of my projects use a baselayer that has a WORKDIR set to something else, so I haven't noticed the change, but this is a rather barebone project.

Bingo, thanks @waterkip for investigating! Looking at https://wiki.debian.org/NewInBookworm, I think this is caused by the merged-usr root filesystem layout implementation https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636:

[nix-shell:~]$ docker run --rm -it debian:bookworm bash
Unable to find image 'debian:bookworm' locally
bookworm: Pulling from library/debian
fc9ce7290e7e: Pull complete
Digest: sha256:9f76a008888da28c6490bedf7bdaa919bac9b2be827afd58d6eb1b916e1e5918
Status: Downloaded newer image for debian:bookworm
root@1a3583529000:/# ls -l
total 48
lrwxrwxrwx   1 root root    7 Jul 25 00:00 bin -> usr/bin
drwxr-xr-x   2 root root 4096 Jul 14 16:00 boot
drwxr-xr-x   5 root root  340 Aug  2 15:02 dev
drwxr-xr-x   1 root root 4096 Aug  2 15:02 etc
drwxr-xr-x   2 root root 4096 Jul 14 16:00 home
lrwxrwxrwx   1 root root    7 Jul 25 00:00 lib -> usr/lib
drwxr-xr-x   2 root root 4096 Jul 25 00:00 media
drwxr-xr-x   2 root root 4096 Jul 25 00:00 mnt
drwxr-xr-x   2 root root 4096 Jul 25 00:00 opt
dr-xr-xr-x 137 root root    0 Aug  2 15:02 proc
drwx------   2 root root 4096 Jul 25 00:00 root
drwxr-xr-x   3 root root 4096 Jul 25 00:00 run
lrwxrwxrwx   1 root root    8 Jul 25 00:00 sbin -> usr/sbin
drwxr-xr-x   2 root root 4096 Jul 25 00:00 srv
dr-xr-xr-x  12 root root    0 Aug  2 15:02 sys
drwxrwxrwt   2 root root 4096 Jul 25 00:00 tmp
drwxr-xr-x  11 root root 4096 Jul 25 00:00 usr
drwxr-xr-x  11 root root 4096 Jul 25 00:00 var

If your build context has a lib then copying into / won't work due to the destination being a symlink (and its also likely the same for {,s}bin as well:)

[nix-shell:~]$ mkdir -p test-sbin-dir/sbin

[nix-shell:~]$ cd test-sbin-dir/

[nix-shell:~/test-sbin-dir]$ cat <<EOF >Dockerfile
> FROM debian:bookworm
> COPY . .
> EOF

[nix-shell:~/test-sbin-dir]$ docker build -t test-sbin .
[+] Building 0.0s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 72B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm         0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 91B                                           0.0s
 => [1/2] FROM docker.io/library/debian:bookworm                           0.0s
 => ERROR [2/2] COPY . .                                                   0.0s
------
 > [2/2] COPY . .:
------
cannot copy to non-directory: /var/lib/docker/overlay2/h3lz230c06nlnp02k3j3n6bjt/merged/sbin

I'm happy to take @Tekki's suggestion to change the default WORKDIR to /usr/src/app or even /app and put out a doc update for everyone - @tianon do we have precedence for other language images that customize WORKDIR?

[waterkip]

python:latest, ruby:latest don't set the workdir, golang:latest sets it /go. gcc:latest, adds an instruction in their docs to add a WORKDIR:

FROM gcc:4.9
COPY . /usr/src/myapp
WORKDIR /usr/src/myapp
RUN gcc -o myapp main.c
CMD ["./myapp"]