A Singer target that writes data to Google BigQuery.
target-bigquery
works together with any other Singer Tap to move data from sources like Braintree, Freshdesk and Hubspot to Google BigQuery.
(originally found in the Google API docs)
Create a file called config.json
in your working directory, following config.sample.json. The required parameters are the project name project_id
, the dataset name dataset_id
, and table name table_id
.
First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.
target-bigquery
can be run with any Singer Tap, but we'll use tap-fixerio
- which pulls currency exchange rate data from a public data set - as an example.
These commands will install tap-fixerio
and target-bigquery
with pip and then run them together, piping the output of tap-fixerio
to target-bigquery
:
› pip install target-bigquery tap-fixerio
› tap-fixerio | target-bigquery -c config.json
INFO Replicating the latest exchange rate data from fixer.io
INFO Tap exiting normally
If you're using a different Tap, substitute tap-fixerio
in the final command above to the command used to run your Tap.
It is recommended to use target-bigquery
with a service account.
target-bigquery
will be executed.GOOGLE_APPLICATION_CREDENTIALS
environment variable on the machine, where the value is the fully qualified path to client_secrets.jsonIt should be possible to use the oAuth flow to authenticate to GCP as well:
target-bigquery
will attempt to open a new window or tab in your default browser. If this fails, copy the URL from the console and manually open it in your browser.target-bigquery
to access your Google BigQuery table.The data will be written to the table specified in your config.json
.
Copyright © 2018 RealSelf, Inc.