Release notes 24.3

This is the third major version of Dataedo in 2024! Read on to find out what new features and improvements we implemented.

New homepage in Portal

We want to deliver value to our users from the very first moment they enter our Portal. We decided to display a number of widgets on the first screen they see - the homepage.

Users will now see:

• Object to review - a section available for users with edit rights, showing a summary of what objects they should have a look at according to workflows.
• Community updates - a widget that collects all the community comments in one place.
• Newly published - displaying a list of newly published objects according to workflows. This list shows objects published during the previous 30 days.
• Recently viewed - a section that reflects browsing history within the Dataedo Portal.
• Bookmarked - in 24.3 we introduce a possibility to bookmark objects you're interested in. You'll be able to see them all on this widget.
• Frequently visited - a list of most often visited objects while browsing Dataedo Portal.

Data Lineage improvements in Portal

Source Level

In the Portal, you can analyze lineage from the Source Level. It represents the connection between data sources. When a data source contains more than 20 objects, they will be grouped by object type with a counter. Click on the object will highlight connection.

For further clarification, the diagram only shows objects that are linked to other data sources.

Manual and generated flows distinguished

When a flow is defined manually by the user in the Dataedo Desktop, it is rendered as a darker and thicker line. Auto-generated flows are shown as solid lines.

Process script in details panel

Double-clicking on a flow line allows you to preview the process script:

Additionally, when a column is transformed into another, the transformation is highlighted in the script as shown below:

Transformation details

When a column flow has any transformation, it is displayed on the diagram as a small dot. When the user hovers over it, the transformation details are displayed.

Easier navigation

It is possible to easily navigate to higher layer diagram. You can click arrow up button to redirect:

• From Column level to Object level
• From Object level to Data source level

Save to PNG

It is now possible to download the Lineage diagram as PNG file. The downloaded image is going to look just the same as the one visible in the browser. So you would need to show and hide these objects you want to see.

Show all columns

Data Lineage diagrams with any column flow defined, have a possibility to click Show all columns to preview all object columns. If a column does not have a flow, it is displayed in grey.

Data Quality Insights

As a first step of introducing Data Quality in Dataedo we are presenting a dashboard with data allowing you to take action: empty columns and tables, missing data for unique columns or ones which row count has rapidly increased or decreased.

Profiling would be required in order to present the data.

Dashboards

We added a number of dashboards to help you have the overview of the entire system. In 24.3 the access to these dashboards is limited to users with edit rights to the entire repository.

The data is refreshed every 1 hour, if a user wants to see up-to-date information they need to click on Refresh button and all the data will be fetched from that moment. Refresh button is next to report's name or at the bottom of each widget, if only a single widget should be refreshed.

System dashboard

We added a new dashboard collecting and showcasing information about Dataedo's repository. We display its size (overall and divided by module with its history in time), the number of keywords, number of tables divided by category, monthly changes made by each user, version installations and more.

Metadata Scanning Dashboard

It is a summary of imports and import changes. It displays various data about data imported in the last 90 days like source types, connector, specific data source. It also lets you know what import steps went fine and which ones failed.

Data Catalog Dashboard

Showcases the amount of objects in the catalog. They are grouped by types and sources.

Reports Usage Dashboard

This dashboard aims to present most popular reports based on statistics downloaded from Power BI. There is also a second tab presenting unused reports, based on the same metric.

Deleted Objects Dashboard

This dashboard groups deleted objects by various categories - by data source, object type.

Schema changes Dashboard

This dashboard collects schema changes from all over your repository. The data is presented on a chart showing changes over time and a grid displaying changes within last 90 days.

Data profiling Dashboard

Collects data and presents profiled tables and non-tabular objects grouped by data source.

Reference Data Dashboard

Lists reference data statistics and lookups with values to be reviewed (accepted or rejected).

Glossary Dashboard

Lists glossary entries grouped by type and glossary they're in.

Steward Hub suggestions in Portal

This year we introduced a Steward Hub, which role is to help and point out what could be done to enhance the documentation. In 24.2 We added first tasks connected with workflows in Portal. In 24.3 we've implemented all the suggestions previously available in Dataedo Desktop into Portal.

Refreshing the suggestions is a manual action, you need to click the refresh button, located on the left side navigation:

Link objects to domains

This list contains objects in your repository, which are not linked to any Business Domain nor their nested areas. Data domains have a corresponding list in their section.

You can review these objects, assign them (or bulk assign) to a selected domain or reject these suggestions that aren't useful.

To be able to see objects on this list you need to have edit permissions for these objects and edit permission on at least one Domain.

Verify terms suggestions

We added a list with suggestions for new terms.

Each suggestion is based on recurring names of columns from your reports. These lists are available to check and they are filtered by edit permissions.

Assign owner

Within Data Roles section we have an Assign Owner list. Owner is any Custom Field of user type (meaning a user from your application can be assigned to it). If you have multiple user type Custom Fields, all of them will be listed as a separate view to take an action.

We will list these objects with nobody assigned to a user type Custom Field, suggesting you to take an action and assign someone.

It is possible to bulk assign users to multiple objects.

Verify Primary Key suggestions

Data Model section contains two types of suggestions - the first one is verifying the suggestions for Primary Keys.

The list contains following information:

• what table PK was created for and based on of what column,
• additional information about the column,
• suggested name for new PK (it's possible to change it on hover).

To be able to see a suggestion, you need to have an edit permission for the table to which the suggestion applies to.

Create Primary Key manually

The second suggestion type of Data Model section is creating Primary Keys manually.

This view contains table-like objects without primary key and containing at least one column. If the suggestion is valid, click on Add PK and a popup with more information will appear.

Bookmarking objects in Portal

There are objects users visit more often than the others. In order to shorten the time to find them we introduced the ability to bookmark these objects.

Later on you can find all the bookmarked objects on the homepage in Bookmarked widget.

Multiple tasks of the same type in Scheduler

In 24.2 we allowed only one task of a specific type. From 24.3 it's possible to add more than one task - e.g. two import changes tasks for a specific data source.

New connectors

• Fivetran connector
• SAP BusinessObjects BI connector
• Qlik Sense On-Premise connector
• Sharepoint Lists connector
• IBM z/OS Custom Connector

Fivetran connector

Dataedo 24.3 introduces a connector for Fivetran. Fivetran is an ELT that supports more than 500 sources. Its most important feature is simplicity, after logging in the source and destination warehouse, all you have to do is select which tables you want to move, and Fivetran will take care of the rest.

Fivetran connector supports column-level data lineage for sources and destinations that are also available in Dataedo. Example supported sources/destinations:

• Salesforce
• Snowflake
• SQLServer

Example not supported sources/destinations:

• Facebook Ads
• Airtable
• Asana

You can learn more about this connector here.

SAP BusinessObjects BI connector

SAP BusinessObjects BI is a suite of front-end applications designed for business intelligence (BI) that helps organizations perform data reporting, analysis, and visualization.

Objects you can document in Dataedo:

• Documents
• Reports
• Universes
• Free hand SQL queries
• Connections

We also support an automated column-level data lineage extraction for this tool. You can read more details here.

Qlik Sense On-Premise connector

Qlik Sense is a popular BI application that lets you analyze, visualize, and present business data. In previous versions, we already had support for a Qlik Sense SaaS. Now we add support for an On-Premise Qlik BI variant as well.

Supported objects:

• Analytics apps
• Apps tables
• Data files
• Connections

This connector automatically gathers column-level data lineage between Analytic apps and App tables and object-level lineage between App tables and Dataset or Source tables.

You can read more details here.

Sharepoint Lists connector

SharePoint Lists are widely used for collaboration and data storage, and with this new connector, you can easily capture and document the structure of your lists, ensuring better organization and understanding of your SharePoint data assets. You can also use those lists in a data lineage flow as well.

Read more details here.

IBM z/OS Custom Connector

IBM z/OS is an operating system specifically designed for IBM's zSeries mainframes, which are specialized machines built for high-performance and large-scale enterprise computing.

Dataedo will document following objects from IBM Db2 for z/OS:

• Tables
• Views
• Procedures
• Functions

Connectors improvements

• Importing basic table statistics
• Power BI reports usage
• Azure Service Principal authentication
• SQL Parser and Data Lineage improvements in Dataedo Desktop
• PostgreSQL procedures and functions parsing
• Snowflake connector improvements
• Snowflake Enterprise Lineage - full column-level lineage within Snowflake instance
• Lineage improvements for SSIS
• Databricks improvements
• Power BI improvements

Importing basic table statistics

Dataedo 24.3 introduces a new import option: "Import Row and Statistics". When enabled, this enchances you tables' metadata with additional statistics, including:

• Row count
• Last user query
• Number of user queries
• Last user update
• Number of user updates
• Last load time

This setting is available for connectors below:

• SQL Server
• MySQL
• PostgreSQL
• Oracle
• Redshift
• Snowflake

(Dataedo Portal)

(Dataedo Desktop)

Power BI reports usage

Power BI connector is now able to import number of views and unique viewers for reports. You can view most popular and least popular reports in Reports Usage dashboard in Dataedo Portal.

Azure Service Principal authentication

Dataedo 24.3 introduces a new authentication method for some Azure connectors. You can now use Azure Service Principal in the following connectors:

• Azure Data Factory
• Power BI
• SharePoint Lists

This method allows you to import without user interaction so that the import of the connectors mentioned above can be scheduled.

SQL Parser and Data Lineage improvements in Dataedo Desktop

You are able to view the data lineage from contants and functions in the Dataedo Desktop lineage diagram:

Automatic data lineage is created for Sql Server Agent jobs/steps using parser.

PostgreSQL procedures and functions parsing

Dataedo Desktop now supports parsing PostgreSQL procedures and functions. Read more details here.

Snowflake connector improvements

New objects supported in Snowflake connector:

• Iceberg tables
• Streams with column-level data lineage
• Sequences

We also improved performance in retrieving PK/UQ/FK constraints.

You can read more details here.

Snowflake Enterprise Lineage - full column-level lineage within Snowflake instance

The Snowflake Enterprise Lineage Connector is a powerful feature that tracks column-level data lineage within your Snowflake instance. This connector allows you to import queries (up to one year old) along with the data lineage they produce, enhancing the overall data governance and traceability of your Snowflake environment.

The Snowflake Enterprise Lineage Connector supplements the standard Snowflake Connector by adding advanced lineage tracking at the query level. It provides a detailed view of data flows between objects, making it a critical tool for data governance, compliance, and auditing.

You can read more details here.

Lineage improvements for SSIS

Dataedo will create lineage to file objects referenced in variables.

Databricks improvements

New objects supported in Databricks connector:

• External locations (Google Cloud, AWS, Azure, Cloudfare) as linked sources,
• Foreign keys.

Also we added support for CMD/Scheduler import. You can read more details here.

Data lineage improvements:

• Data lineage for objects that use external locations,
• Column-level data lineage for tables, views and external tables,
• Now data lineage is built automatically across different catalogs.

You can read more details here.

Power BI improvements

• Enhanced object and column-level lineage building through DAX script parsing, improving lineage quality and accuracy.
• Added support for column-level lineage from files.
• Introduced column-level lineage for Dataflows.
• Enabled column-level lineage from Dataflows to Dataset Tables.
• Column-level lineage now works even when the dataset is in another workspace.
• Power BI relationships can now be imported.

Bulk add Glossary Entries including Custom Fields

From 24.3 it is possible to bulk add Terms with Custom Fields. This is an improvement of existing "Bulk add Entries" feature.

It is important that we also change positions of some columns. From the left there are now term title and type columns instead of parent title and type. Customers who use this feature, in 24.3 will have to change their template. There is a "Copy template" button in the ribbon that can help.

Ability to load definitions from excel file into glossary

From now on it is be possible to use the load definitions for glossary entries. Users will be able to fill Custom Fields and definitions for existing terms easily.

Security enhancements

We have addressed the following security vulnerabilities: CVE-2024-30105, CVE-2019-0657, CVE-2019-0981, CVE-2019-0980, CVE-2018-8292, CVE-2021-24112, CVE-2024-27086, CVE-2024-35255, CVE-2019-0820, CVE-2023-21893, CVE-2022-34716, CVE-2024-38095, GHSA-qrmm-w75w-3wpx, CVE-2024-29992. These fixes improve the overall security and stability of the system.

License update

We have also updated our license within this version. We added a sentence in point 5.4: Activation of the License will occur automatically 60 days from the date of the invoice.