"skaffold deploy --force=false" still force updates deployments

[nashyeung]

Expected behavior

The command "skaffold deploy --force=false" should not recreate pods in the deployment.

Actual behavior

The command "skaffold deploy --force=false" recreates pods in the deployment.

Information

• Skaffold version: v0.37.1
• Operating system: ubuntu 18.04
• Kubernetes environment: minikube v1.3.1 (vm-driver=none)
• Kubectl version: v1.14.0
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta13
kind: Config

deploy:
  kubectl:
    manifests:
    - nginx-deployment.yaml

• Contents of nginx-deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80

Steps to reproduce the behavior

1. Put the 2 files above skaffold.yaml and nginx-deployment.yaml into the same folder
2. Inside that folder, run skaffold deploy --force=false --verbosity=debug
3. See these 3 lines at the end of log output, the nginx deployment is updated, nginx pods are recreated

DEBU[0000] Running command: [kubectl --context minikube apply -f - --force]
deployment.apps/nginx configured
Deploy complete in xxxxx ms

4. Try skaffold deploy -v debug without specifying the --force option, which defaults to false. Still see kubectl ... -f - --force in debug logs, and nginx pods being recreated
5. Try all other falsy values for the --force parameter, e.g. f, FALSE, 0. Still see kubectl ... -f - --force in the debug logs
6. Try skaffold dev/run/debug. --force works well, i.e. no more kubectl ... --force in debug logs (although pods are still recreated, should be caused by Allow users to override the run-id #2752 I think)
7. Try SKAFFOLD_FORCE=false skaffold deploy -v debug. No more kubectl ... --force this time.

[giladsh1]

@nashyeung I ran into the same issue...
see #2687 and #2752 on how to fix.

tl;dr - to prevent skaffold from recreating resources which didn't actually change,
add -l skaffold.dev/run-id=SOMETHING to your skaffold command.

[balopat]

Thank you for filing @nashyeung and @giladsh1 for jumping in!
@nashyeung can you confirm that adding the runid override solves the problem?

I'm thinking though whether we should turn off run-id labelling when --force=false?

[nashyeung]

Tried using the build "770ccce". Confirmed the run-id fix works.
All these use cases work as expected, the deployment remains unchanged.

• skaffold deploy -l skaffold.dev/run-id=asdf
• skaffold (dev or run or debug) -l skaffold.dev/run-id=asdf --force=false

Still a bit confused by the debug log, since I always see Running command: [kubectl --context minikube apply -f - --force], even when I run skaffold deploy -l skaffold.dev/run-id=asdf --force=false.
I'd expect --force=false to remove the --force option from the kubectl command.
Not a real problem though.

And thanks @giladsh1 and @balopat for your help in my k8s journey. (a ton of new stuff to learn!)
I'll use the latest build for now, and wait for the fix #2687 / #2752 to be released.

[taisph]

The label override doesn't work for me on 0.37.1. It still adds a random UUID to run-id and redeploys everything. Works in 0.38.0.

skaffold deploy -n test --profile gcp --build-artifacts skaffold-build.json --label skaffold.dev/run-id=stop-force-deploy

[balopat]

Thank you for the update! Closing!

[balopat]

Still a bit confused by the debug log, since I always see Running command: [kubectl --context minikube apply -f - --force], even when I run skaffold deploy -l skaffold.dev/run-id=asdf --force=false.
I'd expect --force=false to remove the --force option from the kubectl command.
Not a real problem though.

I think we should still look into this.