Issue while uploading data to an existing bigquery table

[VikramTiwari]

1. OS type and version : macOS, v10.12
2. Python version and virtual environment information python --version : Python 2.7.12
3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

gax-google-logging-v2==0.8.3
gax-google-pubsub-v1==0.8.3
gcloud==0.18.3
google-api-python-client==1.5.4
google-apitools==0.5.4
google-cloud-dataflow==0.4.1
google-gax==0.12.5
googleapis-common-protos==1.3.5
grpc-google-logging-v2==0.8.1
grpc-google-pubsub-v1==0.8.1
grpcio==1.0.1rc1

1. Stacktrace if available

Traceback (most recent call last):



  File "readAndUploadFacebookReport.py", line 135, in <module>

    main()

  File "readAndUploadFacebookReport.py", line 128, in main

    load_transformed_data_to_bq('fbschema.json', transform_filename)

  File "readAndUploadFacebookReport.py", line 83, in load_transformed_data_to_bq

    job = table.upload_from_file(readable, source_format='text/csv', skip_leading_rows=1)

  File "/usr/local/lib/python2.7/site-packages/gcloud/bigquery/table.py", line 913, in upload_from_file

    return client.job_from_resource(json.loads(response_content))

  File "/usr/local/lib/python2.7/site-packages/gcloud/bigquery/client.py", line 119, in job_from_resource

    config = resource['configuration']

KeyError: 'configuration'

1. Steps to reproduce
2. Tried pushing data into an existing table on BQ using table.upload_from_file on a csv file with proper schema and other configurations. Code breaks on that line as well.
3. Code example

with open(filepath, 'rb') as readable:
        job = table.upload_from_file(readable, source_format='text/csv', skip_leading_rows=1, write_disposition="WRITE_APPEND")

[dhermes]

@VikramTiwari This seems to be the same as #2098. Is there something stopping you from upgrading to version 0.20.0?

[VikramTiwari]

Well, could be. I just did upgrade and tried uninstall and re-install, yet I am stuck on 0.18.3. Here's logs.

$ python -m pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /usr/local/lib/python2.7/site-packages
Requirement already up-to-date: grpcio>=1.0rc1 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: oauth2client>=2.0.1 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: googleapis-common-protos in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: protobuf!=3.0.0.b2.post1,>=3.0.0b2 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: google-gax<0.13dev,>=0.12.3 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: grpc-google-pubsub-v1<0.9dev,>=0.8.0 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: httplib2>=0.9.1 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: gax-google-logging-v2<0.9dev,>=0.8.0 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: gax-google-pubsub-v1<0.9dev,>=0.8.0 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: grpc-google-logging-v2<0.9dev,>=0.8.0 in /usr/local/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: six in /Library/Python/2.7/site-packages/six-1.10.0-py2.7.egg (from gcloud)
Requirement already up-to-date: futures>=2.2.0 in /usr/local/lib/python2.7/site-packages (from grpcio>=1.0rc1->gcloud)
Requirement already up-to-date: enum34>=1.0.4 in /usr/local/lib/python2.7/site-packages (from grpcio>=1.0rc1->gcloud)
Requirement already up-to-date: pyasn1-modules>=0.0.5 in /usr/local/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Requirement already up-to-date: pyasn1>=0.1.7 in /usr/local/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Requirement already up-to-date: rsa>=3.1.4 in /usr/local/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Requirement already up-to-date: setuptools in /usr/local/lib/python2.7/site-packages (from protobuf!=3.0.0.b2.post1,>=3.0.0b2->gcloud)
Requirement already up-to-date: future>=0.15.2 in /usr/local/lib/python2.7/site-packages (from google-gax<0.13dev,>=0.12.3->gcloud)
Requirement already up-to-date: ply==3.8 in /usr/local/lib/python2.7/site-packages (from google-gax<0.13dev,>=0.12.3->gcloud)

[dhermes]

See the README, after 0.18.3 we renamed gcloud to google-cloud.

[VikramTiwari]

Ah! My bad. I was aware of the changes on Node.js side. Upgraded to latest one.

gapic-google-logging-v2==0.10.1
gapic-google-pubsub-v1==0.10.1
gax-google-logging-v2==0.8.3
gax-google-pubsub-v1==0.8.3
google-api-python-client==1.5.4
google-apitools==0.5.4
google-cloud==0.20.0
google-cloud-bigquery==0.20.0
google-cloud-bigtable==0.20.0
google-cloud-core==0.20.0
google-cloud-dataflow==0.4.1
google-cloud-datastore==0.20.1
google-cloud-dns==0.20.0
google-cloud-error-reporting==0.20.0
google-cloud-language==0.20.0
google-cloud-logging==0.20.0
google-cloud-monitoring==0.20.0
google-cloud-pubsub==0.20.0
google-cloud-resource-manager==0.20.0
google-cloud-storage==0.20.0
google-cloud-translate==0.20.0
google-cloud-vision==0.20.0
google-gax==0.14.1
googleapis-common-protos==1.3.5
grpc-google-iam-v1==0.10.1
grpc-google-logging-v2==0.10.1
grpc-google-pubsub-v1==0.10.1
grpcio==1.0.1rc1

Will test it tomorrow morning and update the issue if it resolves or otherwise. Thanks @dhermes 👍

[dhermes]

Thanks

[tseaver]

@VikramTiwari I believe #2365 resolved this issue. Please reopen with further details if I am wrong.

[VikramTiwari]

This was solved with updating version/library (google-cloud). Thanks for closing!