Piotr Kononow
I am proud to present you a new Dataedo: 23.1. This is the first release of the next generation of our product. We have a number of new features, improvements and new metadata connectors and automations.
From this release we introduced new numbering system - year and release number. "23" stands for 2023, ".1" is first major release this year. As usual, we are determined to deliver 3-4 new releases every year. Each bringing new and exciting capabilities.
Before you upgrade
Before you upgrade please make sure to see the following:
- Upgrade requirements - this release requires upgrade of the following:
- Repository
- Web Catalog
- Desktop
- End of support for file repositories - from 23.1 Dataedo will only support server repository. This is due to our focus on development of collaborative data catalog, rather than stand alone application. See section below for details.
- End of support for NetSuite - due to technical constraints we had to close support for NetSuite connector in Desktop. See section below for details and workarounds.
New connectors and improvements
Tableau connector [new capabilities]
In Dataedo 23.1, we are adding a new type connector for Tableau, one of the most popular BI tools.
Dataedo imports following structures:
- Workbooks and Sheets
- Database Tables
- Custom SQL Tables
- Virtual Connection Tables
- Embedded and Published Data Sources
Dataedo builds column-level data lineage automatically for:
- flows between Workbooks and Data Sources
- flows between Data Sources and Tables
- flows between Tables and source databases that have been imported in Dataedo
Users can migrate the database imported by the Tableau Data Model connector to the Reporting type without losing data.
Export of descriptions from Dataedo to Tableau!
For users who have the Tableau Data Management Addon, there is also the option to export descriptions from Dataedo with a list of linked terms for columns in Database Tables to Tableau. Exported descriptions are inherited by Data Source fields. In the Tableau Web editor, they are always visible to the user, while in the Tableau Desktop editor, exported descriptions are visible only for Published Data Sources.
Learn more about Tableau connector
Power BI [new capabilities]
We overhauled our Power BI connector significantly. Improvements include:
- Creation of column-level lineage between reports, datasets and sources
- Importing of dataflows
- Premium workspace and XMLA Endpoint are no longer required, now you can connect to Power BI Pro
Learn more about Power BI connector
SSRS connector [new]
We created a new BI connector for SQL Server Reporting Services.
Connector imports the following objects:
- Reports
- Datasets - embedded and shared
Learn more about SSRS connector
Snowflake [new capabilities]
We extended Snowflake connector. Now it is capable of importing:
- Snowpipes (added in 10.4)
- Stages
- Snowpipe data lineage
Learn more about Snowflake connector
Redash connector [new]
Dataedo 23.1 can connect to Redash and import Dashboards, Visualizations, Queries and create lineage automatically.
Learn more about Redash connector
Automated column-level lineage of views from SQL parsing
In Dataedo 10.4 we automated data lineage creation for SQL views using our own parsing technology.
In 23.1 we extended support for the following dialects:
- MySQL (and forks, such as MariaDB)
- PostgreSQL (supports also Redshift)
- PL/SQL (Oracle)
Since 10.4 Dataedo supports also:
- Transact-SQL (SQL Server, Azure SQL, Azure Synapse)
We are working on support for:
- Snowflake
Data Lineage improvements
Apart from extraction and automation of data lineage documentation mentioned above, we also
- Dropped limitations of manual lineage creation - now all object types can read and write data (including tables) and have unlimited number of processes
Data Profiling improvements
Data Profiling feature got an overhaul. We redesinged UI to make it easier to run, save, refresh and clear profiling, describe column, and improved security by disabling of profiling of classified columns.
Edit descriptions and classifications directly in profiling window
When you are browsing column profile - its distribution, top or sample values - and you learn about the column, you can now describe and classify column directly in the same window.
Profiling of classified columns is disabled
To improve security, profiling data, other than distribution, of classified columns is disabled.
Definition of which classification labels should be disabled for profiling is configurable in new Settings option.
Web Catalog
Web Catalog new design, once again
We are constantly working on improving Web Catalog design and user experience. We present you the next version of our catalog:
Notifications
Dataedo 23.1 has a new notification system that aggregates notifications from community and edits. User gets notifications from commetns/edits of object that he/she folows, or when they get mentioned.
Mentions
Now, in community posts you can mention other users:
They will receive a notification of mention:
Your profile
Now, you can edit your profile name and picture. It will be visible in the community and on User type fields.
Web Catalog improvements
- Performance improvements - we did refactoring of some of Web Catalog code and now some of the pages will open much, much faster
- Security improvements:
- Secured Stored XSS security vulnerability
New custom field types
In 23.1 we added a few types of custom fields:
- Tag
- User
- Date
- Color
Tag
Tag custom field works just like you'd expect it to - users can create and assign labels to various elements of the catalog.
Then you can use those tags for searching.
User
This is a useful new custom field type that allows you to assign a person to any asset or element. You could create fields such as Owner, Steward, SME, etc.
You can then use to filter by specific person.
Date
Date custom field allows you to add restrict users to provide valid dates in specific fields.
Color
Color field allows you to categorize assets and elements in the catalog to predefined color categories, similarly to MS Outlook.
New object types
Till now, Dataedo had the following main object types in the catalog:
- Reports
- Reports
- Data assets
- Tables
- Views
- Dataset
- SQL Query - introduced in 10.4, allows you to save your favorite SQL scripts
- Code
- Procedures
- Functions
In 23.1 we made the types more granular, to better represent objects that we import and catalog, and we added some new types:
- Reports
- Reports
- Dashboards
- Visualizations
- Data assets
- Tables
- Views
- Dataset
- SQL Query - introduced in 10.4, allows you to save your favorite SQL scripts
- Graph Nodes - taken out of the Tables folder, for documentation of graph databases
- Graph Edges - taken out of the Tables folder, for documentation of graph databases
- Search Indexes - taken out of the Tables folder, used to document Search Indexes, i.e. from Elasticsearch
- Cubes - taken out of the Tables folder, OLAP cubes
- Dimensions - taken out of the Tables folder, Dimensions from OLAP databases
- Data Types - might be used to document user-defined data types
- Sequences - will be used to extract and document database sequences
- Topics - in the future used to document Kafka topics
- Application Lists - allow you to document lists in your applications, such as CRM
- Application Forms - allow you to document forms in your applications, such as CRM
- Entities
- Sources - will be used for ETL documentation
- Destinations - will be used for ETL documentation
- Code
- Procedures
- Functions
- Packages - taken out of the Procedures folder, will be used to document Oracle packages
- ETL Programs - new object type used to document ETL packages and data pipelines
- Scripts/Programs - new object type that represents generic script or program
- Tasks/Jobs - new object type that will be used to import and represent database tasks/jobs
- SQL Scripts - new object type, similar to SQL Query, that will allow you to document SQL code or snippet (that does not return a dataset)
- APIs - new object type that represents API method
Note: Your data and imports will stay in the folders as previously. In our next release we will migrate all the data to new folders.
FK Relationship tester - more sources
In Dataedo 10.4 we created a feature that helps you validate foreign key relationship in Desktop.
In 23.1 we extended support for the following connectors:
- PostgreSQL
- MySQL
- Oracle
- Redshift
- Aurora
Since 10.4 Dataedo supports also:
- SQL Server
- Azure SQL
- Snowflake
We are working on support for:
- Azure Synapse
AD Kit: Active Directory Integration
With Dataedo 23.1 we're introducing ADKit - command line application that allows to synchronize users between Active Directory and Web Catalog. It can be setup to automatically add, delete, and update users from AD group to chosen group in Web Catalog. It should help with users management (especially in larger organizations), as administrators won't have to create users manually.
End of support for NetSuite connector
Starting with Dataedo version 23.1, Dataedo will no longer support the NetSuite connector. It will not be possible to add NetSuite as a new data source and update changes.
Workaround:
Leave one copy of Dataedo Desktop 10.4 and 10.4 repository, that you will be using to import NetSutie schema. After you import, copy data into interfacing tables in the 23.x repository and use interface tables import.
End of support for file repositories
From 23.1 Dataedo will only support server repository. This change is due to our focus on development of collaborative data catalog, rather than a stand alone application.
Migrating to server repository
It is no longer possible to migrate file repository to a server repository. If you're still using a file repository please make sure you've migrated your repository to a server one before upgrading Dataedo application.
Merging repositories
It is now possible to merge two repositories into one and merge its contents. You can select one of the chosen repositories and click Merge repositories button.
After selecting the second repository to be merged you will be informed on the amount of objects that are going to appear after merging. Please double check if these numbers are correct and make sure you made a backup.
After merging the target repository will contain of all the objects from both repositories.