Jib 'cache checking' is too slow when there's many modules (eg microservices)

[iirekm]

Many times ./gradlew ... is called - this is overkill even with Gradle daemon!

Much faster is running Jib manually:

   custom: 
        buildCommand: ./gradlew :project:path:jibDockerBuild --image $IMAGE

Single call to ./gradlew should be enough for whole build.

[balopat]

Thanks for the issue!
Can you tell us a bit more about this? Is this happening when you call skaffold dev and rebuild on file changes?

cc @GoogleContainerTools/java-tools

[iirekm]

During scaffold run. For every microservice a separate ./gradlew is started, which is too costy. I even confirmed this using strace.

[loosebazooka]

So there was a bug on this, but it should be fixed: #3483

What version of skaffold are you on? But it's quite possible you're talking about something else and I misunderstood.

[iirekm]

The recent version, 1.3.1. For 4 Spring Boot modules, Skaffold runs ./gradlew at least 4 times.

[loosebazooka]

@iirekm we treat each module as its own build. But it's also not clear to me how running using a custom buildcommand is faster, can you elaborate on that?

[iirekm]

I have no idea why it's faster, but indeed it was faster on my machine:

   custom: 
        buildCommand: ./gradlew :project:path:jibDockerBuild --image $IMAGE

x4 times was faster than running the jib's cache checks x4 times

[loosebazooka]

I'm not sure I understand what you mean by cache checks? Can you provide your skaffold.yaml with jib configured and with your custom builder configured on 4 different artifacts?

[iirekm]

deploy/skaffold.yaml:

apiVersion: skaffold/v1
kind: Config

build:
  artifacts:
    - image: module1
      context: ..
      jib:
        project: applications:module1
    - image: module2
      context: ..
      jib:
        project: applications:module2
    - image: module3
      context: ..
      jib:
        project: applications:module3
    - image: module4
      context: ..
      jib:
        project: applications:module4
   
  local:
    push: false # for now I use k3s over Docker, so this is enough

deploy:
  helm:
    releases:
      - name: envelo
        chartPath: .
        values:          
          module1.image: module1
          module2.image: module2
          module3.image: module3
          module4.image: module4          
        wait: true

    flags:
      upgrade:
        - --install # needed for Helm 3

And then:

$ strace -ff skaffold run 2>&1 | grep execve | grep gradlew     
[pid 30189] execve("/home/.../workspace/.../gradlew", ["/home/.../workspace/.../g"..., "_skaffoldFailIfJibOutOfDate", "-Djib.requiredVersion=1.4.0", ":applications:module1:_jibSkaffo"..., "-q"], 0xc0004caa00 /* 78 vars */ <unfinished ...>

And the same for every module.

[loosebazooka]

I don't actually see how running gradlew on your own is any faster? In order to inject the $IMAGE param correctly for each sub-project, they need to each be run as separate invocations. Perhaps it's not clear to me what you're suggesting. Could you describe in some extended detail how you expect this work?

[dgageot]

What's happening here is that Skaffold's cache mechanism needs to get each module's list of dependencies in order to compute a digest and compare with what it has already built before.
This currently runs gradle for each module, not matter wether it's a cache hit or miss.
This is slow indeed.

When you switch to a custom artifact, you get faster results because it then doesn't ask gradle about the dependencies. It assumes the whole workspace is a dependency and computes a digest for the files contained in it. That can be much faster.

@GoogleContainerTools/java-tools-build Any idea what we could do to improve that behaviour?