Documenting and cataloging Amazon QuickSight

Amazon QuickSight is a scalable, serverless, and machine learning-powered business intelligence (BI) service provided by AWS. It enables users to create and share interactive dashboards, perform ad-hoc analysis, and gain insights from their data through a web-based interface. QuickSight integrates seamlessly with various AWS data sources and offers advanced features like natural language querying, predictive analytics, and embedded analytics for custom applications.

Catalog and Documentation

Amazon Quicksight Dashboards

Overview

On the overview page, you will se basic information about the dashboard such as name, created date and last updated date.

Visualisations and fields

During the import process, Dataedo will document the following dashboard's elements (only for Amazon Quicksight Enterprise edition):

• Sheets
• Visualisations
• Calculated fields

On the following, sample dashboard, you can see a few visualisations inside Sheet 1 such as Revenues vs Goals (which uses Revenue calculated field), YTD Revenues by Service Line and Segment and Revenues by Customer Segment.

The dashboard will be documented by Dataedo in the following way:

• Revenue calculated field - field no. 1 with computed type and formula in Constraints column
• Sheet 1 sheet - field no. 2 with Sheet type as parent of visualisations
• Revenues vs Goals visualisation - field no 2.1 with visualisation type in Type column (in that case Line Chart Visual)
• YTD Revenues by Service Line and Segment visualisation - field no 2.2 with visualisation type in Type column (in that case Pivot Table Visual)
• Revenues by Customer Segment visualisation - field no 2.1 with visualisation type in Type column (in that case Bar Chart Visual)

Amazon Quicksight Analyses

Overview

On the overview page, you will se basic information about the analysis such as name, created date and last updated date.

Visualisations and fields

During the import process, Dataedo will document the following analysis's elements (only for Amazon Quicksight Enterprise edition):

• Sheets
• Visualisations
• Calculated fields

On the following, sample analysis, you can see a few visualisations inside Sheet 1 such as Revenues vs Goals (which uses Revenue calculated field), YTD Revenues by Service Line and Segment and Revenues by Customer Segment.

The analysis will be documented by Dataedo in the following way:

• Revenue calculated field - field no. 1 with computed type and formula in Constraints column
• Sheet 1 sheet - field no. 2 with Sheet type as parent of visualisations
• Revenues vs Goals visualisation - field no 2.1 with visualisation type in Type column (in that case Line Chart Visual)
• YTD Revenues by Service Line and Segment visualisation - field no 2.2 with visualisation type in Type column (in that case Pivot Table Visual)
• Revenues by Customer Segment visualisation - field no 2.1 with visualisation type in Type column (in that case Bar Chart Visual)

Amazon Quicksight Datasets

Datasets, along with their fields, are imported as datasets. The logical and physical layers (including calculated fields, information about SQL queries, and the tables and files joined to create datasets) under Quicksight datasets are imported as dataset tables.

Overview

The overview tab contains basic informating regarding dataset.

For the following, sample dataset Revenue which uses one S3 file datasource named REVENUE DS. There will be created in Dataedo two objects - dataset Revenue and dataset tabe REVENUE DS (if dataset is created by joining several physical tables/files or datasources all underlying objects will be created as dataset tables)

Fields

Dataedo will document all dataset's fields and calculated fields with their data type and formulas.

Scripts

In the case where dataset is built from custom sql query, this script is saved in the documentation.

Amazon Quicksight Datasources

Quicksight datasources are imported as Dataedo linked sources and used as foundation for building lineage to external documentations.

Data Lineage

Amazon Quicksight connector supports creating Data lineage to enable tracking data flows. Imported lineage differs between Amazon Quicksight editions.

For Amazon Quicksight Enterprise:

• column level lineage between dashboards and analyses for calculated fields and visualizations. (for Amazon Quicksight Standard edition there is only object level lineage)
• column level lineage between datasets and analyses for calculated fields. (for Amazon Quicksight Standard edition there is only object level lineage)
• column level lineage for all fields between datasets and the underlying logical and physical layers (imported as Dataedo Dataset Tables). The logical and physical layers consist of other datasets, physical tables, files, and custom SQL queries joined together to create a dataset.
• column level lineage between external data source columns (S3 files, Redshift, Postgres, SQL Server, MySQL database tables) and fields in the logical and physical layers of datasets (imported as Dataedo Dataset Tables). This lineage comes from imported metadata and SQL parsing (for custom SQL queries). External datasource has to be documented in Dataedo and the documentation has to reside in the same repository as the Amazon Quicksight documentation.

Data Lineage limitations

• For Amazon Amazon Quicksight Standard edition there is object level lineage (without column-level lineage) between datasets and analyses and between analyses and dashboards
• Lineage for dataset files uploaded to quicksight manually is not supported
• Lineage for S3 files whose manifest.json file is uploaded to quicksight is not supported. The manifest.json file must reside on S3 bucket to which used IAM user has s3:GetObject action enabled
• If IAM user used to import Amazon Quicksight into Dataedo doesn't have s3:GetObject action enabled on bucket on which manifest.json of S3 datasource is stored lineage will be not created
• Importing of columns and column level lineage is not supported for datasets created with "Upload a file" Amazon QuickSight functionality (it is not supported by Amazon QuickSight API)

Required access level

To connect Amazon QuickSight to Dataedo you'll need to have IAM user with programmatic access enabled and policy with following actions enabled on all resources:

• quicksight:ListAnalyses
• quicksight:DescribeAnalysis
• quicksight:ListDashboards
• quicksight:DescribeDashboard
• quicksight:ListDataSets
• quicksight:DescribeDataSet
• quicksight:DescribeDataSource

Amazon Quicksight Enterprise users can also enable actions to make available importing calculated fields and visualizations of analyses and dashboards:

• quicksight:DescribeAnalysisDefinition
• quicksight:DescribeDashboardDefinition

If you are using S3 datasources IAM user has to have s3:GetObject action enabled on each bucket with manifest.json file.

Connecting to Amazon Quicksight

Amazon Quicksight IAM user configuration

To connect to Amazon Quicksight you'll need to create IAM user that has permission to get QuickSight objects.

To set up that user:

• Log in to your AWS Console and then go to the IAM management console
• Create a new IAM user by clicking Users tab and then Create user
• Specify user details and click Next
• Select Attach policies directly and then click Create policy
• Select JSON option and paste in the following policy and click Next

Policy for Amazon Quicksight Standard Edition:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "quicksight:DescribeDashboard",
                "quicksight:ListAnalyses",
                "quicksight:ListDataSources",
                "quicksight:ListDashboards",
                "quicksight:DescribeAnalysis",
                "quicksight:ListDataSets",
                "quicksight:DescribeDataSet",
                "quicksight:DescribeDataSource"
            ],
            "Resource": "*"
        }
    ]
}

Policy for Amazon Quicksight Enterprise Edition (if there is no need for importing analyses and dashboards contents and column level lineage you can use policy from Amazon Quicksight Standard Edition)

Policy for Amazon Quicksight Enterprise Edition:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "quicksight:DescribeDashboard",
                "quicksight:ListAnalyses",
                "quicksight:ListDataSources",
                "quicksight:ListDashboards",
                "quicksight:DescribeAnalysis",
                "quicksight:ListDataSets",
                "quicksight:DescribeDataSet",
                "quicksight:DescribeDataSource",
                "quicksight:DescribeAnalysisDefinition",
                "quicksight:DescribeDashboardDefinition"
            ],
            "Resource": "*"
        }
    ]
}

• Fill policy details and click Next
• Select created policy and click Next
• Review user and policy details and click Create User
• Click Users on left sidebar
• Click on just created user name
• Under ARN label there is AWS Account ID number which will be needed in Dataedo import arn:aws:iam::YOUR_AWS_ACCOUNT_ID:user/quicksight_testy

• Click Create access key
• Select Other
• Click Create access key
• Access Key ID and Secret Access Key are shown please copy them to use in Dataedo import

When you have IAM number, access key and secret access key the last thing to check is AWS region of your Amazon Quicksight. To check it please login to Amazon Quicksight portal - after successfull login the region is displayed in URL for example for URL: https://eu-west-1.quicksight.aws.amazon.com the region is eu-west-1

Dataedo connection

To connect to Amazon Quicksight select Add source -> New connection. On the connectors list select Amazon Quicksight (beta).

Fill AWS Account ID, Access Key ID and Secret Access Key and select Region. If you wish to save Secret Access Key for future imports check Save Secreat Access Key checkbox.

Click Connect and go through the import process.

Outcome

Specification

Imported metadata

Dataedo reads following metadata from Amazon Quicksight