Release notes 24.2

We're proud to let you know that the second major version in 2024 is available! Read on to find out what we have prepared this time.

Scheduler

In 24.2 we introduced a Scheduler. Its primary goal is to allow users to schedule some crucial operations so that they are performed automatically and periodically at a specific time. It will also allow to perform these operations on dedicated, external machines so that they don't burden and block resources of users computers. Possible operations that can be scheduled in 24.2:

• import changes,
• refresh data profiling.

In version 24.2, Scheduler will be available only in Dataedo Desktop.

The dedicated machines for performing scheduled tasks will require Dataedo Desktop (installed or the portable version), so they have to be the Windows machines. The actual performer of scheduled tasks will be a Windows service, called a Dataedo Agent, setup by users with a dedicated, console-based installer.

The first view after opening Scheduler is the summary of scheduled tasks. A task is a definition of periodically repeated operation, characterized by a type (e.g. import connector), connection parameters (e.g. login and password), options and scheduling details (e.g. every day at 8 pm). In the upper part of the form you can see all of your databases with checked these operations that have already been scheduled. The list below contains a detailed list of tasks. After double-clicking on a row, detailed information about the task will appear on the right.

In the Executions view you can see all the executions that have already taken place. An execution is a single instance of the task at a specific time and day. You can filter them using dedicated filters located above the grid. After double-clicking on an execution, detailed information about it will appear on the right, including a history, connection parameters and details.

It is possible to add many tasks for many data sources (each task will be executed sequentially, one after another).

Workflows

In 24.1 we introduced native manual objects' statuses. It was a foundation for workflows we implemented in 24.2.

Right now you're able to configure your own workflow: define its statuses, determine who can set them and who should take some action once an object is set to a certain status. Moreover, every status has the property Visible for viewers, as a result objects in certain status are either visible or hidden for users without edit rights. Keep it in mind that all not published objects will now not be available to the Viewers and Explorers in your organization. You need to first publish them to be visible again.

Read more about workflows configuration here.

There is a new role, Approver, with the exact same permissions as Data Steward. We did this to make it easier to distinguish between those responsible for acceptance and those who document.

We added a new navigation element - Steward Hub. Every user with edit rights will be able to see a list of objects assigned to them. This is where you can check what is assigned to you and massively change the status of objects to the next in the workflow.

Workflows are available only if you have a Dataedo Plan subscription, read more about a license in Portal here.

Public API

The Public API is an interface designed to help our users integrate Dataedo with other tools. In 24.2 we're releasing the first version of the API. While it primarily allows for data fetching, it also supports updating and creating certain objects.

We use HTTP POST and HTTP GET methods. Every endpoint used in the Public API requires a special token, which is used in the ADKit tool.

Here is a brief summary of what is currently possible and what will be possible in the near future.

Data Lineage improvements

We implemented a number of improvements and new features for the Data Lineage in our Dataedo Portal.

Data lineage for columns

We added Data Lineage for columns. The appearance and behavior is the same as for object-level diagrams.

Flow details panel

You can now check flow details panel between objects or between columns - it displays additional information about processor, scripts, transformations and more. Simply click on the line between objects to see it in a side panel

Show and hide processors

We added ability to hide or show processors, you can switch between them as you like. By processors we mean objects that only transform data, such as procedures or functions. They are hidden by default. Here you can compare the same Dataset when hiding processors:

and showing them:

Full screen mode for lineage

You can explore Data Lineage on expanded and full screen view to make it easier for browsing the lineage.

All of the above are also available in the Dataedo Desktop since from now on we're embedding the view from Portal into Desktop. It is required to configure Dataedo Portal first and login to Portal in the Desktop app.

Dashboards

We introduced a new section - Dashboards. We started with a first step, which is Usage Dashboard. We showcase active users in our application by month and the most active users. The first chart is visible to all the users with edit rights, the second one is available only to the admin user.

Steward Hub

In 24.2 we added new features to Steward Hub:

• Linking objects to Business Domains and Data Domains (including their Areas). Objects without any link to Domains or Areas will appear in the suggestions.
• Objects with unassigned Owners (Custom Fields with User type) will show up, so that they can be defined.
• Adding Primary Keys to tables with no PK created.

Qlik Sense Cloud connector

We received multiple requests to introduce a Qlik connector so in Dataedo 24.2 it's here! You can now connect and harvest metadata including data lineage from Qlik BI.

You can read more details about the scope and connecting instructions in the docs.

(Dataedo Portal)

(Dataedo Desktop)

(Dataedo Desktop)

REST API (OpenAPI) connector

As REST APIs can be a crutial part of data flows we created connector that documents structure of your endpoints from an OpenAPI specification in YAML or JSON format.

You can find more details here.

Oracle E-Business Suite connector

Dataedo now lets you document EBS database including tables, views, procedures and dependencies. It also automaticlly shows column-level data lineage from tables to views. Thanks to that you can understand your complex EBS system much better and faster.

Read more in the docs

(Dataedo Portal)

Apache Atlas integration

For the first time, we introduced a connector to the different data catalog. You can gather metadata from many Apache Atlas instances into Dataedo. This approach follows the idea of catalog of catalogs.

Read more about how to connect and the scope here

Amazon QuickSight connector

In this relase we added support for Amazon QuickSight BI. You can now pull sheets, datasets with fields and calculated fields, the logical and physical layers of datasets (including calculated fields, SQL queries, and joined tables/files), and data sources from it. We also support automated data lineage on object and column level.

Check more details here

Connector improvements

Snowflake

• Importing tasks as jobs
• Importing dynamic tables
• Importing materialized views

SSIS

• Automatic data lineage support for ODBC connections
• Automatic data lineage for "SQL Server Destination"

SSAS Tabular

• Support for SSAS Tabular 2017
• Data lineage based on PowerQuery

PowerBI

• Dataset tables are now imported as seperate objects (improved readability and data lineage)

Azure Data Factory and Synapse Pipelines

• Datasets are no longer shown on a data lineage diagram
• Runtime data lineage. We scan run logs to represent flows more accurately
• Loop Activitity enhancements. This improvement is particularly beneficial for activities that run within loops or are executed multiple times with different parameters

Profiling history

We now display not only the current profiling results, but also the history of changes in profiling data. This allows to analyze and verify whether it's correct and how it changed over time.

In column's overview you can see more details about the selected one and redirect to profiling to see more information:

In the profiling tab you can check the detailed history of how the data changed

Filtering, sorting and pagination

We focused on improving our grids in the Dataedo Portal. These are the list views used for browsing columns, parameters, terms etc. We improved issues regarding filtering, added sorting and pagination.

Import process improvements

In the import process, we removed the step that showed the list of objects to be imported. It has been replaced with a list of schemas, along with the types of objects to import:

Node 'Future schemas' allows you to select the types of objects that will be selected when a new schema appears on the list during subsequent imports.

For database types that do not support schemas, only the list of types to be imported will be shown.

It is still possible to define your advanced filters by checking the 'Advanced filters' checkbox. This change eliminates the long waiting period for objects to be shown during import.

AI generated descriptions in Desktop

From 24.2 we display AI generated descriptions in Dataedo Desktop for tables and views (with their columns) and terms. The generation is done in the Dataedo Portal, so if there is no AI generated description, we enable one-click generation and redirection to the Portal.

New built-in classification (PIPEDA)

We added a new built-in classification - Personal Information Protection and Electronic Documents Act (PIPEDA), a Canadian regulation regarding personal information

UX improvements

Editing in Portal

Edit and view modes are no longer in Portal. Right now editing any field requires hovering over selected field and clicking a pen icon, which switches the field to edit mode. Editing is available only to these users with required edit permissions, just like it was before this change.

Column switcher

Switching between columns doesn't require navigating back to the table and its columns. It can be easily done via clicking an arrow pointing down next to column's name and then clicking on a desired column name.

We also unified inputs and behaviours, fixed bugs and typos in the Portal.