Libraries and tools for interoperability between Apache Hadoop related open-source software and Google Cloud Platform.
The Google Cloud Storage connector for Hadoop enables running MapReduce jobs directly on data in Google Cloud Storage by implementing the Hadoop FileSystem interface. For details, see the README.
The Google BigQuery connector for Hadoop MapReduce enables running MapReduce jobs on data in BigQuery by implementing the InputFormat & OutputFormat interfaces. For more details see the documentation
This connector is deprecated and was removed - it's recommended to use Apache Bahir instead.
To build the connector for specific Hadoop version, run the following commands from the main directory:
# with Hadoop 2 and YARN support: ./mvnw -P hadoop2 clean package # with Hadoop 3 and YARN support: ./mvnw -P hadoop3 clean package
In order to verify test coverage for specific Hadoop version, run the following commands from the main directory:
# with Hadoop 2 and YARN support: ./mvnw -P hadoop2 -P coverage clean verify # with Hadoop 3 and YARN support: ./mvnw -P hadoop3 -P coverage clean verify
The Cloud Storage connector JAR can be found in
gcs/target/. The BigQuery
connector JAR can be found in
Maven group ID is
com.google.cloud.bigdataoss and artifact ID for Cloud
Storage connector is
gcs-connector and for BigQuery connectors is
To add a dependency on one of the connectors using Maven, use the following:
<dependency> <groupId>com.google.cloud.bigdataoss</groupId> <!-- Cloud Storage: --> <artifactId>gcs-connector</artifactId> <version>hadoop2-2.1.3</version> <!-- or, for BigQuery: --> <artifactId>bigquery-connector</artifactId> <version>hadoop2-1.1.3</version> </dependency>
On Stack Overflow, use the tag
for questions about the connectors in this repository. This tag receives
responses from the Stack Overflow community and Google engineers, who monitor
the tag and offer unofficial support.