Release notes 23.1

Applies to: Dataedo 24.x (current) versions, Article available also for: 23.x

I am proud to present you a new Dataedo: 23.1. This is the first release of the next generation of our product. We have a number of new features, improvements and new metadata connectors and automations.

From this release we introduced new numbering system - year and release number. "23" stands for 2023, ".1" is first major release this year. As usual, we are determined to deliver 3-4 new releases every year. Each bringing new and exciting capabilities.

Before you upgrade

Before you upgrade please make sure to see the following:

  1. Upgrade requirements - this release requires upgrade of the following:
    • Repository
    • Web Catalog
    • Desktop
  2. End of support for file repositories - from 23.1 Dataedo will only support server repository. This is due to our focus on development of collaborative data catalog, rather than stand alone application. See section below for details.
  3. End of support for NetSuite - due to technical constraints we had to close support for NetSuite connector in Desktop. See section below for details and workarounds.

New connectors and improvements

Tableau connector [new capabilities]

In Dataedo 23.1, we are adding a new type connector for Tableau, one of the most popular BI tools.

Image title

Dataedo imports following structures:

  • Workbooks and Sheets
  • Database Tables
  • Custom SQL Tables
  • Virtual Connection Tables
  • Embedded and Published Data Sources

Dataedo builds column-level data lineage automatically for:

  • flows between Workbooks and Data Sources
  • flows between Data Sources and Tables
  • flows between Tables and source databases that have been imported in Dataedo

Users can migrate the database imported by the Tableau Data Model connector to the Reporting type without losing data.

Export of descriptions from Dataedo to Tableau!

For users who have the Tableau Data Management Addon, there is also the option to export descriptions from Dataedo with a list of linked terms for columns in Database Tables to Tableau. Exported descriptions are inherited by Data Source fields. In the Tableau Web editor, they are always visible to the user, while in the Tableau Desktop editor, exported descriptions are visible only for Published Data Sources.

Image title

Learn more about Tableau connector

Power BI [new capabilities]

We overhauled our Power BI connector significantly. Improvements include:

  1. Creation of column-level lineage between reports, datasets and sources
  2. Importing of dataflows
  3. Premium workspace and XMLA Endpoint are no longer required, now you can connect to Power BI Pro

Learn more about Power BI connector

SSRS connector [new]

We created a new BI connector for SQL Server Reporting Services.

Image title

Connector imports the following objects:

  1. Reports
  2. Datasets - embedded and shared

Learn more about SSRS connector

Snowflake [new capabilities]

We extended Snowflake connector. Now it is capable of importing:

  1. Snowpipes (added in 10.4)
  2. Stages
  3. Snowpipe data lineage

Learn more about Snowflake connector

Redash connector [new]

Dataedo 23.1 can connect to Redash and import Dashboards, Visualizations, Queries and create lineage automatically.

Image title

Learn more about Redash connector

Automated column-level lineage of views from SQL parsing

In Dataedo 10.4 we automated data lineage creation for SQL views using our own parsing technology.

Image title

In 23.1 we extended support for the following dialects:

  • MySQL (and forks, such as MariaDB)
  • PostgreSQL (supports also Redshift)
  • PL/SQL (Oracle)

Image title

Since 10.4 Dataedo supports also:

  • Transact-SQL (SQL Server, Azure SQL, Azure Synapse)

We are working on support for:

  • Snowflake

Data Lineage improvements

Apart from extraction and automation of data lineage documentation mentioned above, we also

  1. Dropped limitations of manual lineage creation - now all object types can read and write data (including tables) and have unlimited number of processes

Data Profiling improvements

Data Profiling feature got an overhaul. We redesinged UI to make it easier to run, save, refresh and clear profiling, describe column, and improved security by disabling of profiling of classified columns.

Edit descriptions and classifications directly in profiling window

When you are browsing column profile - its distribution, top or sample values - and you learn about the column, you can now describe and classify column directly in the same window.

Image title

Profiling of classified columns is disabled

To improve security, profiling data, other than distribution, of classified columns is disabled.

Image title

Definition of which classification labels should be disabled for profiling is configurable in new Settings option.

Image title

Web Catalog

Web Catalog new design, once again

We are constantly working on improving Web Catalog design and user experience. We present you the next version of our catalog:

Image title

Notifications

Dataedo 23.1 has a new notification system that aggregates notifications from community and edits. User gets notifications from commetns/edits of object that he/she folows, or when they get mentioned.

Image title

Mentions

Now, in community posts you can mention other users:

Image title

They will receive a notification of mention:

Image title

Your profile

Now, you can edit your profile name and picture. It will be visible in the community and on User type fields.

Image title

Web Catalog improvements

  1. Performance improvements - we did refactoring of some of Web Catalog code and now some of the pages will open much, much faster
  2. Security improvements:
    • Secured Stored XSS security vulnerability

New custom field types

In 23.1 we added a few types of custom fields:

  • Tag
  • User
  • Date
  • Color

Tag

Tag custom field works just like you'd expect it to - users can create and assign labels to various elements of the catalog.

Image title

Then you can use those tags for searching.

Image title

User

This is a useful new custom field type that allows you to assign a person to any asset or element. You could create fields such as Owner, Steward, SME, etc.

Image title

You can then use to filter by specific person.

Image title

Date

Date custom field allows you to add restrict users to provide valid dates in specific fields.

Image title

Color

Color field allows you to categorize assets and elements in the catalog to predefined color categories, similarly to MS Outlook.

Image title

New object types

Till now, Dataedo had the following main object types in the catalog:

  • Reports
    • Reports
  • Data assets
    • Tables
    • Views
    • Dataset
    • SQL Query - introduced in 10.4, allows you to save your favorite SQL scripts
  • Code
    • Procedures
    • Functions

In 23.1 we made the types more granular, to better represent objects that we import and catalog, and we added some new types:

  • Reports
    • Reports
    • Dashboards
    • Visualizations
  • Data assets
    • Tables
    • Views
    • Dataset
    • SQL Query - introduced in 10.4, allows you to save your favorite SQL scripts
    • Graph Nodes - taken out of the Tables folder, for documentation of graph databases
    • Graph Edges - taken out of the Tables folder, for documentation of graph databases
    • Search Indexes - taken out of the Tables folder, used to document Search Indexes, i.e. from Elasticsearch
    • Cubes - taken out of the Tables folder, OLAP cubes
    • Dimensions - taken out of the Tables folder, Dimensions from OLAP databases
    • Data Types - might be used to document user-defined data types
    • Sequences - will be used to extract and document database sequences
    • Topics - in the future used to document Kafka topics
    • Application Lists - allow you to document lists in your applications, such as CRM
    • Application Forms - allow you to document forms in your applications, such as CRM
    • Entities
    • Sources - will be used for ETL documentation
    • Destinations - will be used for ETL documentation
  • Code
    • Procedures
    • Functions
    • Packages - taken out of the Procedures folder, will be used to document Oracle packages
    • ETL Programs - new object type used to document ETL packages and data pipelines
    • Scripts/Programs - new object type that represents generic script or program
    • Tasks/Jobs - new object type that will be used to import and represent database tasks/jobs
    • SQL Scripts - new object type, similar to SQL Query, that will allow you to document SQL code or snippet (that does not return a dataset)
    • APIs - new object type that represents API method

Note: Your data and imports will stay in the folders as previously. In our next release we will migrate all the data to new folders.

FK Relationship tester - more sources

In Dataedo 10.4 we created a feature that helps you validate foreign key relationship in Desktop.

Image title

In 23.1 we extended support for the following connectors:

  • PostgreSQL
  • MySQL
  • Oracle
  • Redshift
  • Aurora

Since 10.4 Dataedo supports also:

  • SQL Server
  • Azure SQL
  • Snowflake

We are working on support for:

  • Azure Synapse

AD Kit: Active Directory Integration

With Dataedo 23.1 we're introducing ADKit - command line application that allows to synchronize users between Active Directory and Web Catalog. It can be setup to automatically add, delete, and update users from AD group to chosen group in Web Catalog. It should help with users management (especially in larger organizations), as administrators won't have to create users manually.

AD Kit installation

End of support for NetSuite connector

Starting with Dataedo version 23.1, Dataedo will no longer support the NetSuite connector. It will not be possible to add NetSuite as a new data source and update changes.

Workaround:

Leave one copy of Dataedo Desktop 10.4 and 10.4 repository, that you will be using to import NetSutie schema. After you import, copy data into interfacing tables in the 23.x repository and use interface tables import.

End of support for file repositories

From 23.1 Dataedo will only support server repository. This change is due to our focus on development of collaborative data catalog, rather than a stand alone application.

Migrating to server repository

It is no longer possible to migrate file repository to a server repository. If you're still using a file repository please make sure you've migrated your repository to a server one before upgrading Dataedo application.

Merging repositories

It is now possible to merge two repositories into one and merge its contents. You can select one of the chosen repositories and click Merge repositories button.

Image title

After selecting the second repository to be merged you will be informed on the amount of objects that are going to appear after merging. Please double check if these numbers are correct and make sure you made a backup.

Image title

After merging the target repository will contain of all the objects from both repositories.