skaffold fails to sync with helm

[cflynn07]

Expected behavior

static files sync to container without a rebuild when updated on disk

Actual behavior

skaffold fails to sync files, prints warn log

Information

• Skaffold version: v0.23.0
• Operating system: macOS 10.14 (18A391)
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta3
kind: Config
build:
  artifacts:
  - image: cflynnus/blog
    context: .
    sync:
      '**/*.html': "/root"
      '**/*.md': "/root"
      '**/*.css': "/root"
      '**/*.png': "/root"
deploy:
  helm:
    releases:
      - name: "blog"
        chartPath: "helm"

• error message from skaffold dev -v debug:

INFO[0309] files modified: [templates/home.html]
DEBU[0310] Checking base image golang:1.11.5-alpine3.8 for ONBUILD triggers.
DEBU[0310] Checking base image alpine:latest for ONBUILD triggers.
DEBU[0310] Found dependencies for dockerfile: [.]
DEBU[0310] Skipping excluded path: node_modules
DEBU[0310] Skipping excluded path: README.md
DEBU[0310] Skipping excluded path: kubernetes
Syncing 1 files for cflynnus/blog:392e972d392cdd136404b52ae4e31ce490cabace4ade88c485dc2debc8dc9b9b
INFO[0310] Copying files: map[templates/home.html:/root/templates/home.html] to cflynnus/blog:392e972d392cdd136404b52ae4e31ce490cabace4ade88c485dc2debc8dc9b9b
WARN[0310] Skipping deploy due to sync error: copying files: didn't sync any files
Watching for changes every 1s...

• complete repository with helm chart/templates and skaffold.yaml
  https://github.com/cflynn07/myblog/tree/dc5115181f030e769ae192f9ecf6aecb8dcec8d2

Steps to reproduce the behavior

1. skaffold dev -v debug
2. update static template file /templates/home.html with trivial change

[laander]

Might be a duplicate of #1668 although reports in the other issue doesn't mention any errors (Skipping deploy due to sync error: copying files: didn't sync any files)

[cflynn07]

I'm able to use skaffold with this project by using kubectl instead of helm; it would just be nice to get it working with helm.

[priyawadhwa]

Hey @cflynn07 I just merged #1709 which may have solved this problem -- could you see if you still get this error with the lastest released version v0.24.0?

[cflynn07]

Still seeing an issue

• checked out the code in my repo at dc5115181f030e769ae192f9ecf6aecb8dcec8d2
• ran skaffold dev
• Made a trivial change to a file
• skaffold dev exited with an error

skaffold dev
WARN[0000] config version (skaffold/v1beta3) out of date: upgrading to latest (skaffold/v1beta6)
Generating Tag for [cflynnus/blog]...
Starting build...
Found [docker-for-desktop] context, using local docker daemon.
Building [cflynnus/blog]...
Sending build context to Docker daemon  34.31MB
Step 1/8 : FROM golang:1.11.5-alpine3.8 as builder
 ---> a0d0176aa927
Step 2/8 : WORKDIR /go/src/app
 ---> Using cache
 ---> 81e84baa853b
Step 3/8 : COPY . .
 ---> Using cache
 ---> 6d4ccbab1eaa
Step 4/8 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
 ---> Using cache
 ---> 51a8d3efe70a
Step 5/8 : FROM alpine:latest
 ---> 3f53bb00af94
Step 6/8 : WORKDIR /root
 ---> Using cache
 ---> a3d5f0e9cf0a
Step 7/8 : COPY --from=builder /go/src/app/ .
 ---> Using cache
 ---> a6c626e4a0ce
Step 8/8 : CMD ["./main"]
 ---> Using cache
 ---> bc3b2a672b4a
Successfully built bc3b2a672b4a
Successfully tagged cflynnus/blog:dc51151
Build complete in 1.086752908s
Starting test...
Test complete in 5.319µs
Starting deploy...
Error: release: "blog" not found
Helm release blog not installed. Installing...
No requirements found in helm/charts.
NAME:   blog
LAST DEPLOYED: Tue Mar  5 17:46:44 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME       TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)       AGE
blog--app  LoadBalancer  10.103.104.183  localhost    80:32303/TCP  0s

==> v1/Deployment
NAME       DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
blog--app  1        1        1           0          0s

==> v1beta1/Ingress
NAME          HOSTS                      ADDRESS  PORTS  AGE
test-ingress  cflynn.us,cflynn-blog.com  80, 443  0s

==> v1/Pod(related)
NAME                        READY  STATUS    RESTARTS  AGE
blog--app-75dcf6dbd4-crf2w  0/1    Init:0/1  0         0s


Deploy complete in 660.121915ms
Watching for changes every 1s...
Cleaning up...
release "blog" deleted
Cleanup complete in 178.160281ms
FATA[0018] calling final callback: sync: retrieving working dir for cflynnus/blog:bc3b2a672b4a98b942d35a3fc32c24a5d8dee423b1f9f75c2a33e3bdda2193c1: retrieving remote config: MANIFEST_UNKNOWN: "manifest unknown"

---

Log output when running with skaffold dev -v debug

Cleaning up...
DEBU[0041] Executing template &{envTemplate 0xc420199800 0xc420b0c480  } with environment map[SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.JRbXop83Np/Listeners XPC_SERVICE_NAME:0 OLDPWD:/Users/caseyflynn LESS:-R DISABLE_AUTO_TITLE:true _:/usr/local/bin/skaffold COLORFGBG:15;0 XPC_FLAGS:0x0 LANG:en_US.UTF-8 ITERM_SESSION_ID:w0t0p2:9E0A5FCF-195F-4DF4-B89B-CC1F8FACA0AD LC_CTYPE:en_US.UTF-8 LSCOLORS:Gxfxcxdxbxegedabagacad TERM_SESSION_ID:w0t0p2:9E0A5FCF-195F-4DF4-B89B-CC1F8FACA0AD ITERM_PROFILE:Default SHELL:/bin/zsh COLORTERM:truecolor TMPDIR:/var/folders/94/zfcfqpmx3g71tck0tfxgwmr40000gn/T/ LOGNAME:caseyflynn PAGER:less PWD:/Users/caseyflynn/workspace/myblog TERM_PROGRAM:iTerm.app PATH:/Users/caseyflynn/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/MacGPG2/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/caseyflynn/.antigen/bundles/robbyrussell/oh-my-zsh/lib:/Users/caseyflynn/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/git:/Users/caseyflynn/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/pip:/Users/caseyflynn/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/lein:/Users/caseyflynn/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/command-not-found:/Users/caseyflynn/.antigen/bundles/Dbz/kube-aliases:/Users/caseyflynn/.antigen/bundles/zsh-users/zsh-syntax-highlighting:/Users/caseyflynn/.antigen/bundles/zsh-users/zsh-autosuggestions:/Users/caseyflynn/.antigen/bundles/bhilburn/powerlevel9k:/Users/caseyflynn/go/bin TERM_PROGRAM_VERSION:3.2.7 TERM:xterm-256color SHLVL:1 KEYTIMEOUT:1 Apple_PubSub_Socket_Render:/private/tmp/com.apple.launchd.Im0nMW4Ki2/Render HOME:/Users/caseyflynn USER:caseyflynn __CF_USER_TEXT_ENCODING:0x0:0:0]
DEBU[0041] Running command: [helm --kube-context docker-for-desktop delete blog --purge]
release "blog" deleted
Cleanup complete in 193.772735ms
FATA[0041] calling final callback: sync: retrieving working dir for cflynnus/blog:634b4449105864b4a6b56a09a863d6fc242d37c0b0861eb43923c7ee064be67f: retrieving remote config: MANIFEST_UNKNOWN: "manifest unknown"

[dee-me-tree-or-love]

Same issue when using kubectl for deployment.
Running skaffold v0.24.0 with minikube version: v0.33.1 with --vm-driver kvm2.

• ran skaffold dev -v debug
• made a trivial change
• getting same error as @cflynn07

...
Successfully built e10d8cf74849
Successfully tagged gcr.io/k8s-skaffold/lotr-quotes:14733c7-dirty
Build complete in 401.08347ms
...
Deploy complete in 1.225985329s
Watching for changes every 1s...
...
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] > lotr-quotes@1.0.0 start /usr/src/app
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] > nodemon index.js
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] 
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] [nodemon] 1.18.10
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] [nodemon] to restart at any time, enter `rs`
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] [nodemon] watching: *.*
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] [nodemon] starting `node index.js`
[lotr-quotes-6dcc8b6dfc-2qxgv lotr-quotes] "lotr_quotes" listening on port 8080!
...
INFO[0026] files modified: [lotr-quotes/index.js]       
DEBU[0027] Checking base image node:10-alpine for ONBUILD triggers. 
DEBU[0027] Found dependencies for dockerfile: [package-lock.json package.json .] 
DEBU[0027] Skipping excluded path: node_modules         
DEBU[0030] Terminating port-forward lotr-quotes-6dcc8b6dfc-2qxgv/lotr-quotes:8080 
Cleaning up...
...
DEBU[0030] Running command: [kubectl --context minikube delete --ignore-not-found=true -f -] 
service "lotr-quotes" deleted
deployment.apps "lotr-quotes" deleted
Cleanup complete in 690.755178ms
FATA[0031] calling final callback: sync: retrieving working dir for gcr.io/k8s-skaffold/lotr-quotes:e10d8cf74849706c1aa4ae979e5c60aac96cff324d9674128b212c86f4a6aedc: retrieving remote config: MANIFEST_UNKNOWN: "Failed to fetch \"e10d8cf74849706c1aa4ae979e5c60aac96cff324d9674128b212c86f4a6aedc\" from request \"/v2/k8s-skaffold/lotr-quotes/manifests/e10d8cf74849706c1aa4ae979e5c60aac96cff324d9674128b212c86f4a6aedc\"."

---

My skaffold.yaml:

apiVersion: skaffold/v1beta6
kind: Config
build:
  artifacts:
  - image: gcr.io/k8s-skaffold/lotr-quotes
    context: ./lotr-quotes/
    sync:
      '**/*.js': . # also tried setting /usr/src/app as the WORKDIR and ./lotr-quotes/

deploy:
  kubectl:
    manifests:
    - ./lotr-quotes/k8s/*

The lotr-quotes manifest:

apiVersion: v1
kind: Service
metadata:
  name: lotr-quotes
  labels: 
    app: lotr-quotes
spec:
  type: NodePort
  selector:
    app: lotr-quotes
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lotr-quotes
  labels: 
    app: lotr-quotes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: lotr-quotes
  template:
    metadata:
      labels:
        app: lotr-quotes
    spec:
      containers:
      - name: lotr-quotes
        image: gcr.io/k8s-skaffold/lotr-quotes
        ports:
        - containerPort: 8080

Dockerfile for the lotr-quotes:

FROM node:10-alpine
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm install

COPY . /usr/src/app
EXPOSE 8080
CMD [ "npm", "start" ]

With .dockerignore:

node_modules
npm-debug.log

[dee-me-tree-or-love]

In fact in my case reverting to v0.23.0 solves the problem...

[dee-me-tree-or-love]

Most likely is related to #1734
And hopefully is transitively fixed 👍

[nkubala]

@dee-me-tree-or-love any chance you could try building from HEAD and seeing if your issue is fixed?

[davidbancor]

Using the latest build from https://storage.googleapis.com/skaffold/builds/latest/skaffold-darwin-amd64 solved this issue for me 👍