Skip to content

feat: add timeout for copying build context on kaniko#8329

Merged
aaron-prindle merged 4 commits intoGoogleContainerTools:mainfrom
hiro-o918:main
Jan 23, 2023
Merged

feat: add timeout for copying build context on kaniko#8329
aaron-prindle merged 4 commits intoGoogleContainerTools:mainfrom
hiro-o918:main

Conversation

@hiro-o918
Copy link
Copy Markdown
Contributor

@hiro-o918 hiro-o918 commented Jan 18, 2023

Related: #7887

Description

I found that uploading build context to kaniko pods sometimes stacks when copying build context on same conditions.
I inspected a pod on stack and found three things.

  • initContainer keeps running
  • /kaniko/buildcontext is empty
  • /tmp/complete file is not created

I did not figure out exact reasons, but it supposed to be caused by creating tarball and uploading to pods.

Current implementation does not has timeout on creating tar, so I implemented this by passing context.Context.

Problem
This implementation assumes that the copying to pod must finish in 5 minutes, otherwise fail.
So We need to pass timeout duration if required.

User facing changes (remove if N/A)

All copying steps to kaniko pod must be finished in 5 minutes.

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 19, 2023

Codecov Report

Merging #8329 (175f45e) into main (290280e) will decrease coverage by 4.61%.
The diff coverage is 54.59%.

❗ Current head 175f45e differs from pull request most recent head d64a772. Consider uploading reports for the commit d64a772 to get more accurate results

@@            Coverage Diff             @@
##             main    #8329      +/-   ##
==========================================
- Coverage   70.48%   65.87%   -4.61%     
==========================================
  Files         515      605      +90     
  Lines       23150    29805    +6655     
==========================================
+ Hits        16317    19634    +3317     
- Misses       5776     8696    +2920     
- Partials     1057     1475     +418     
Impacted Files Coverage Δ
cmd/skaffold/app/cmd/completion.go 13.04% <0.00%> (-1.25%) ⬇️
cmd/skaffold/app/cmd/config/list.go 65.21% <ø> (ø)
cmd/skaffold/app/cmd/config/set.go 88.72% <ø> (ø)
cmd/skaffold/app/cmd/config/util.go 54.28% <ø> (ø)
cmd/skaffold/app/cmd/credits.go 100.00% <ø> (ø)
cmd/skaffold/app/cmd/credits/export.go 0.00% <0.00%> (ø)
cmd/skaffold/app/cmd/deploy.go 40.90% <0.00%> (-12.94%) ⬇️
cmd/skaffold/app/cmd/flags.go 93.00% <ø> (+2.18%) ⬆️
cmd/skaffold/app/cmd/generate_pipeline.go 60.00% <ø> (ø)
cmd/skaffold/app/cmd/inspect_modules.go 65.00% <ø> (ø)
... and 416 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@hiro-o918
Copy link
Copy Markdown
Contributor Author

hiro-o918 commented Jan 20, 2023

  • I fixed lint error
  • modify configuration to check timeout of tar_test which failed on only windows
    • But I'm not sure the reason, do you have any idea about this?

@aaron-prindle
Copy link
Copy Markdown
Contributor

Thanks for the PR here @hiro-o918! In looking at out CI/CD tests it seems the test TestAddFileToTarTimeout is failing:

--- FAIL: TestAddFileToTarTimeout (0.01s)
    --- FAIL: TestAddFileToTarTimeout/TestAddFileToTarTimeout (0.01s)
        tar_test.go:302: expected error, but returned none

@hiro-o918
Copy link
Copy Markdown
Contributor Author

hiro-o918 commented Jan 20, 2023

@aaron-prindle
Thank you for checking this PR.
The test fails only on windows, and I do not have any windows machine now.
I'll investigate the why context timeout does not work on windows but if your team knows any clues, please tell me 🙏

@hiro-o918 hiro-o918 marked this pull request as draft January 21, 2023 02:25
@hiro-o918 hiro-o918 force-pushed the main branch 13 times, most recently from 88413b4 to 0c5c67b Compare January 21, 2023 16:21
@hiro-o918 hiro-o918 marked this pull request as ready for review January 21, 2023 16:25
@hiro-o918
Copy link
Copy Markdown
Contributor Author

hiro-o918 commented Jan 21, 2023

Windows error is fixed by increasing the duration of sleeping to wait timeout.
I suppose that It may be caused by difference among OSs to handle go routine.

Copy link
Copy Markdown
Contributor

@aaron-prindle aaron-prindle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the PR here @hiro-o918!

@aaron-prindle aaron-prindle merged commit 1d4f5e2 into GoogleContainerTools:main Jan 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants