added some extra logging for test failures for easier feedback by balopat · Pull Request #1763 · GoogleContainerTools/skaffold

balopat · 2019-03-08T20:25:31Z

This adds some extra logging at the bottom of make test run and saves it in a temporary logfile so that you don't have to manually scroll through issues - as our test output is getting long.

codecov-io · 2019-03-08T20:35:35Z

Codecov Report

Merging #1763 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1763   +/-   ##
=======================================
  Coverage   46.97%   46.97%           
=======================================
  Files         130      130           
  Lines        6252     6252           
=======================================
  Hits         2937     2937           
  Misses       3014     3014           
  Partials      301      301

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 464728c...74116c1. Read the comment docs.

nkubala

👍

tejal29 · 2019-03-08T21:43:42Z

how about dropping the -v flag?

make test
Running go tests...
?   	github.com/GoogleContainerTools/skaffold/cmd/skaffold	[no test files]
?   	github.com/GoogleContainerTools/skaffold/cmd/skaffold/app	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd	(cached)	coverage: 9.0% of statements
ok  	github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd/config	(cached)	coverage: 36.9% of statements
ok  	github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/flags	(cached)	coverage: 52.2% of statements
?   	github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/tips	[no test files]
?   	github.com/GoogleContainerTools/skaffold/cmd/skaffold/man	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/bazel	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/compose	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/getting-started	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/kaniko	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/kaniko-local	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/microservices/leeroy-app	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/microservices/leeroy-web	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/structure-tests	[no test files]
?   	github.com/GoogleContainerTools/skaffold/examples/tagging-with-environment-variables	[no test files]
?   	github.com/GoogleContainerTools/skaffold/hack/new_config_version	[no test files]
?   	github.com/GoogleContainerTools/skaffold/hack/release_notes	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/hack/schemas	(cached)	coverage: 90.6% of statements
ok  	github.com/GoogleContainerTools/skaffold/integration	(cached)	coverage: 0.0% of statements
?   	github.com/GoogleContainerTools/skaffold/integration/examples/bazel	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/compose	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/getting-started	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/kaniko	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/kaniko-local	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/microservices/leeroy-app	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/microservices/leeroy-web	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/structure-tests	[no test files]
?   	github.com/GoogleContainerTools/skaffold/integration/examples/tagging-with-environment-variables	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/apiversion	(cached)	coverage: 100.0% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/bazel	(cached)	coverage: 88.5% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build	(cached)	coverage: 69.3% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/bazel	(cached)	coverage: 18.2% of statements
--- FAIL: TestSetArtifact (0.00s)
    --- FAIL: TestSetArtifact/set_dockerfile_path (0.00s)
        docker_test.go:73: *latest.Artifact differ (-got, +want): {*latest.Artifact}.ArtifactType.DockerArtifact.DockerfilePath:
            	-: "path/to/Dockerfile"
            	+: "path/to/Dockerfile1"
FAIL
coverage: 16.0% of statements
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/docker	0.100s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/kaniko	(cached)	coverage: 3.4% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/kaniko/sources	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/local	(cached)	coverage: 32.8% of statements
time="2019-03-08T13:31:18-08:00" level=warning msg="key-one=value label exists, saving key-one=value as key-one-rand=value to avoid overlap"
--- FAIL: TestPluginBuilderLabels (0.00s)
    --- FAIL: TestPluginBuilderLabels/check_overlapping_labels (0.00s)
        plugin_test.go:104: map[string]string differ (-got, +want): {map[string]string}["key-one-rand"]:
            	-: "value"
            	+: "valu1e"
FAIL
coverage: 12.8% of statements
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/plugin	0.135s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/tag	(cached)	coverage: 82.8% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/color	(cached)	coverage: 82.4% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/config	(cached)	coverage: 93.8% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/constants	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/deploy	(cached)	coverage: 56.2% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/deploy/kubectl	(cached)	coverage: 54.3% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/docker	(cached)	coverage: 71.1% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/event	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/event/proto	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/gcp	(cached)	coverage: 92.0% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/jib	(cached)	coverage: 95.5% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes	(cached)	coverage: 23.1% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/context	(cached)	coverage: 76.9% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/plugin/builders/bazel	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/plugin/builders/docker	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/plugin/environments/gcb	(cached)	coverage: 9.2% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/plugin/shared	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/runner	(cached)	coverage: 61.6% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema	(cached)	coverage: 90.9% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/defaults	(cached)	coverage: 37.5% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/latest	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/util	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1alpha1	(cached)	coverage: 78.1% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1alpha2	(cached)	coverage: 75.0% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1alpha3	(cached)	coverage: 80.8% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1alpha4	(cached)	coverage: 72.7% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1alpha5	(cached)	coverage: 77.8% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1beta1	(cached)	coverage: 72.7% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1beta2	(cached)	coverage: 72.7% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1beta3	(cached)	coverage: 72.7% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1beta4	(cached)	coverage: 72.7% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1beta5	(cached)	coverage: 75.0% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/v1beta6	(cached)	coverage: 75.0% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/sources	[no test files]
time="2019-03-08T13:31:19-08:00" level=info msg="Changed file main.go does not match any sync pattern. Skipping sync"
time="2019-03-08T13:31:19-08:00" level=info msg="Changed file some/other/file does not match any sync pattern. Skipping sync"
--- FAIL: TestNewSyncItem (0.00s)
    --- FAIL: TestNewSyncItem/multiple_sync_patterns (0.00s)
        sync_test.go:290: *sync.Item differ (-got, +want): {*sync.Item}.Image:
            	-: "test:1123"
            	+: "test:123"
FAIL
coverage: 77.0% of statements
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/sync	0.194s
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/sync/kubectl	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/test	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/test/structure	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/update	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/util	(cached)	coverage: 52.4% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/version	(cached)	coverage: 71.4% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/skaffold/warnings	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/watch	(cached)	coverage: 67.9% of statements
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/yamltags	(cached)	coverage: 92.8% of statements
?   	github.com/GoogleContainerTools/skaffold/pkg/webhook/constants	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/webhook/gcs	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/webhook/github	[no test files]
?   	github.com/GoogleContainerTools/skaffold/pkg/webhook/kubernetes	[no test files]
ok  	github.com/GoogleContainerTools/skaffold/pkg/webhook/labels	(cached)	coverage: 100.0% of statements
?   	github.com/GoogleContainerTools/skaffold/testutil	[no test files]
?   	github.com/GoogleContainerTools/skaffold/webhook	[no test files]
Makefile:81: recipe for target 'test' failed
make: *** [test] Error 1

The only reason why i dont like this change is we can't see why test failed with your change.
e.g. On your branch if you purposefully fail a test, the output will be capped and we will need to scroll up again to see the understand why the test failed. With this change, we need to scroll up more now.

*****
***** 'go test' failed (log file: /tmp/tmp.jO9WKvwD89) *****
*****
=== RUN   TestNewSyncItem/no_change_no_sync
=== RUN   TestNewSyncItem/slashes_in_glob_pattern
--- FAIL: TestNewSyncItem (0.00s)
    --- PASS: TestNewSyncItem/match_copy (0.00s)
    --- PASS: TestNewSyncItem/no_tag_for_image (0.00s)
    --- PASS: TestNewSyncItem/multiple_sync_patterns (0.00s)
    --- PASS: TestNewSyncItem/recursive_glob_patterns (0.00s)
    --- FAIL: TestNewSyncItem/sync_all (0.00s)
        sync_test.go:290: *sync.Item differ (-got, +want): {*sync.Item}.Copy["node/index.html"]:
            	-: <non-existent>
--
    --- PASS: TestPerform/client_error (0.00s)
    --- PASS: TestPerform/no_copy (0.00s)
FAIL
coverage: 77.0% of statements
FAIL

vs seeing the whole output.

    --- FAIL: TestNewSyncItem/sync_all (0.00s)
        sync_test.go:290: *sync.Item differ (-got, +want): {*sync.Item}.Copy["node/index.html"]:
            	-: <non-existent>
            	+: "index.html"
            {*sync.Item}.Copy["node/index.html1"]:
            	-: "index.html1"
            	+: <non-existent>

tejal29

Consider dropping -v instead

balopat · 2019-03-09T00:44:14Z

That's awesome feedback @tejal29, thank you!

This reverts commit 8b087c8.

added some extra logging for test failures for easier feedback

8b087c8

balopat requested review from dgageot, nkubala, priyawadhwa and r2d4 as code owners March 8, 2019 20:25

googlebot added the cla: yes label Mar 8, 2019

nkubala approved these changes Mar 8, 2019

View reviewed changes

tejal29 self-requested a review March 8, 2019 21:44

tejal29 suggested changes Mar 8, 2019

View reviewed changes

balopat added 2 commits March 8, 2019 16:44

Revert "added some extra logging for test failures for easier feedback"

1124eda

This reverts commit 8b087c8.

dropping -v

74116c1

dgageot approved these changes Mar 11, 2019

View reviewed changes

dgageot merged commit 6332732 into GoogleContainerTools:master Mar 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added some extra logging for test failures for easier feedback#1763

added some extra logging for test failures for easier feedback#1763
dgageot merged 3 commits intoGoogleContainerTools:masterfrom
balopat:improved_testsh_output

balopat commented Mar 8, 2019

Uh oh!

codecov-io commented Mar 8, 2019 •

edited

Loading

Uh oh!

nkubala left a comment

Uh oh!

tejal29 commented Mar 8, 2019 •

edited

Loading

Uh oh!

tejal29 left a comment

Uh oh!

balopat commented Mar 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

balopat commented Mar 8, 2019

Uh oh!

codecov-io commented Mar 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nkubala left a comment

Choose a reason for hiding this comment

Uh oh!

tejal29 commented Mar 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tejal29 left a comment

Choose a reason for hiding this comment

Uh oh!

balopat commented Mar 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov-io commented Mar 8, 2019 •

edited

Loading

tejal29 commented Mar 8, 2019 •

edited

Loading