Connecting to Google BigQuery

Applies to: Dataedo 23.x versions, Article available also for: 24.x (current), 10.x

This article explains how to connect Dataedo to Google BigQuery and import metadata - tables & columns.

Prerequisities

Service Account

Dataedo connects to Google BigQuery with Service Account. If you don't have one, please read carefully following Google Cloud docs:

  1. Authentication
  2. Understanding service accounts

During creation (or modification) of service account you need to specify roles which will allow Dataedo to document BigQuery database. Dataedo reads INFORMATION_SCHEMA views to collect metadata, hence minimum permissions (roles) that the service account needs to have are:

  • BigQuery Metadadata Viewer - to read metadata,
  • BigQuery Job User - to run a query that reads meatadata.

Create service account

Service account key

To authenticate to Google BigQuery with service account, you need Service Account Key. To get one, you need to find Service Account resource, open Service Account that will be used to connect from Dataedo, go to Keys tab and click Add Key -> Create new Key button. Then select JSON format and download the key.

Create service account key

Cloud Resource Manager API

In order to document BigQuery database, Dataedo needs to access Cloud Resource Manager API which needs to be enabled first. Most likely this feature will be enabled, however if it is disabled, you can activate it by searching Cloud Resource Manager API in Google Marketplace and then clicking Enable button.

Cloud Resource Manager API

Add new connection

To connect to Google BigQuery create new documentation by clicking Add documentation and choosing Database connection.

Add connection

On the Add documentation widnow choose Google BigQuery:

Google BigQuery Add docs

Connection details

Provide connection details:

  • Service Access Key - path to Service Account Key. You can open file explorer to browse for the file by clicking [...] button,
  • Project - select Google Cloud project under which your BigQuery database was created. You can expand list of available projects by clicking [...] button,
  • Dataset - select one or more datasets to document. You can expand list of datasets by clicking [...] button.

Google BigQuery connection details

Importing metadata

When connection was successful Dataedo will read objects and show a list of objects found. You can choose which objects to import. You can also use advanced filter to narrow down list of objects.

BigQuery objects

Confirm list of objects to import by clicking Next.

Next screen allow you to change default name of the documentation under which it will be visible in Dataedo repository.

BigQuery documentation title

Click Import to start the import.

BigQuery import progress

When done close import window with Finish button.

Outcome

Your database schema has been imported to new documentation in the repository.

BigQuery documentation

Google BigQuery support

Google BigQuery support details

Found issue with this article? Comment below
Comments are only visible when the visitor has consented to statistics cookies. To see and add comments please accept statistics cookies.
0
There are no comments. Click here to write the first comment.