What's new in Dataedo

Applies to: 10.x (current) versions, Article available also for: 9.x

Dataedo 10

Data Profiling

Dataedo 10 allows you to discover data stored in the database and review its contents and quality. Data Profiling module is a combination of useful metrics with a friendly User Interface. On top of it, Profiling in Dataedo allows you to peek into the most common, or random data from your tables. Learn more about Data Profiling.

Data Profiling

Part of the improvements in the area of Data Profiling is the fact that we surfaced the ability to peak into Sample Data. Right now you can simply use a handy right-click option on a table to preview its values!

Sample data

Data Lineage

We introduced Data Lineage, in two parts. You can design flows using Dataedo Desktop and browse diagrams using Dataedo Web Catalog. Read more about Data Lineage in Dataedo.

Lineage

In Dataedo 10.3 we also brought the ability to set column-level data lineage.

History of changes in documentation

We introduced the history of changes in descriptions, titles, and Custom Fields. Dataedo will store information about who, when, and how changed the values.

History

Read more about History of changes.

Online licenses and Dataedo Account

This biggest change in the 10 will present itself at the login screen to the Desktop App. You can now log in to Dataedo using your business email. Also, you can seamlessly switch through all your available licenses.

Account

Permissions in the Web Catalog

From version 10.2, accessing any data via Dataedo Web Catalog requires correct permission. This allows detailed customization regarding who will be able to perform what action in the Dataedo Web Catalog.

Actions

Read more about Permissions in Dataedo Web Catalog.

Email Community notifications

We are introducing a feature of notifying users about comments and questions, regarding objects they are following.

Notification

Find out more about Email Notifications in Dataedo Web Catalog

Export and Import with DDL script

You can now export your data structure to data description language (DDL). In other words, you will be able to export the database to a set of statements like CREATE and ALTER. Read more about Export to DDL.

Export DDL

Starting from Dataedo 10.2 you can import your database schema using SQL dump from your database.

DDL

Read more about Importing from DDL script.

Import from interface tables

You can now import your database schema using interface tables, built-in Dataedo.

Interface

Read more about Importing from Interface Tables.

Redesigned Data Classification

We redesigned Data Classification and improved performance or algorithms. You can now classify only a portion of the Data Catalog and configure classification to match your needs.

Image title

We also added a few built-in classifications, so that the current list contains:

  • CCPA - California Consumer Privacy Act
  • FERPA - Family Educational Rights and Privacy Act
  • GDPR - finds columns potentially holding personal data as defined by European GDPR
  • HIPAA - Health Insurance Portability and Accountability Act
  • PCI - Payment Card Industry
  • PII - finds columns PII (Personally Identifiable Information)

Read more about Data Classification.

New metadata connectors

We added a set of new Metadata scanners:

We introduced a way, to import files like JSON, CSV, or XLS directly from:

Learn more about supported databases by Dataedo.

Improvements in existing connectors

We made a few improvements in existing metadata connectors:

  • Amazon Redshift and Azure SQL Database are now supported with Data Profiling.
  • Azure SQL Database multi-factor authentication is now supported.
  • PostgreSQL with better SSL support.

Dataedo 9

Dataedo Web Catalog

Dataedo 9 brings completely new way of interacting with data catalog. 9.1 adds option to edit descriptions.

Data Community in Dataedo Web Catalog

Dataedo 9.1 adds Community features to allow this discussion to occur naturally. Viewer users can now comment on objects and their documentation to point out any required changes in the database or issues with the documentation itself. This and future changes will empower your end-users to become true Data Citizens.

Editing Descriptions in Web Catalog

Sometimes, when browsing through descriptions in the Web Catalog you could encounter information that needs clarification or typos. Before, to correct these issues an editor needed to start Dataedo Desktop and make the changes there. In Dataedo 9.1 you can edit titles, descriptions, and custom field values directly from Dataedo Web.

Authentication with Okta/SAML in Web Catalog

From 9.2 Dataedo Web Catalog can be configured so that users can log in with Okta and possibly with other SAML providers. This simplifies deployment for admins and adoption in the company. It's enough for admins to enable access to Web Catalog in the SAML provider and users are able to log into the portal. 9.2 does not check available licenses. This will functionality be released at a later date.

Learn more about Okta/SAML

Data Classification view in Web Catalog

New Data Classification module added to the Web Catalog showing a summary of column classifications grouped by classification, data domain and classification.

Primary/Unique keys (PK/UK) and Relationships (FK) editor

We are constantly expanding documentation editing capabilities for data stewards. Dataedo 9.2 adds option to define and edit primary and unique keys (PK/UK) and relationships (foreign keys) between tables and other data objects.

Support for Data Lake files

One of the most requested features was an option to document files commonly used in data lakes. In Dataedo 9.1, we've added option to import schema from files. We are constantly extending support for various formats and now the list of sources is the following:

  1. JSON,
  2. XML,
  3. CSV,
  4. Apache Avro,
  5. Apache Parquet,
  6. Apache ORC,
  7. Delta Lake.

Support for NoSQL databases and new connectors

Dataedo 9 adds support for a number of new databases, including NoSQL databases, search engines, and data warehouses:

  1. MongoDB (most popular document database),
  2. Neo4J (most popular graph database),
  3. Elasticsearch (most popular search engine),
  4. Apache Cassandra (most popular wide column database),
  5. Google BigQuery,
  6. Teradata,
  7. Vertica Analytics Platform.

MongoDB Support

Designer extension - manual views, procedures and functions

Till 9.2 objects designer only supported tables. From 9.3, you can design manual views, procedures and functions too.

Copy module/ERD

Now, you can copy entire module with ERD and linked objects.

Option to paste multiple cells

We added ability to paste multiple cells at a time (with multiple values or the same), similarly to MS Excel.

Option to paste one value:

Option to paste and delete multiple values:

Business Glossary in (free) PDF export

From 9.2 free PDF export can also include a Business Glossary. This allows you to share your Business Glossaries to unlimited audience, without the need for the Enterprise Viewer licenses.

Azure Authentication support

This is a much-requested feature from Azure Active Directory users - now, you can connect to Dataedo repository and your Azure/SQL Server databases when importing schema with Azure Authentication.

Azure Authentication Support

Show/hide schema names

Till now, Dataedo decided whether to show schema name in the object name or not based on the DBMS. Many users pointed out that they would like to decide

Now, Dataedo has a smarter algorithm to show schema name – based on DBMS and number of schemas in your database, and it provides option to override this manually.

Azure Authentication Support

Dataedo 8

Dark theme

Dataedo 7

New sources and databases

In Dataedo 7 we added a number of new native connectors and a generic ODBC connection that works with majority of sources.

Native connectors:

  • PostgreSQL
  • IBM Db2
  • Snowflake
  • Amazon Redshift
  • Percona Server for MySQL
  • MySQL 8.x

Business Glossary (Enterprise)

Dataedo Enterprise metadata repository has now the ability to define a common language and understanding of key terms across the organization - architects, developers and business users - in a Business Glossary, and link it to specific data assets (tables or fields).

Manual Data Dictionary

Manual documentations

While we are constantly working on including more data sources in Dataedo, we understand that at any given time some will be unsupported.

To fix this, from Dataedo 7.0.2 you can add create your documentation and tables from within the app.

Manual documentations

Later on, when we add support for your data source, you will be able to just click connect and import the actual structure from your database without losing your descriptions.

Manual tables in existing documentations

Additionally, you can now add tables not existing in your database to the documentation.

Just click the Add table button in the ribbon, or right click the documentation you want to create the table in and you can define your new table.

Manual tables in existing documentations

Schema Change Tracking (Enterprise)

Often, it's important to be able to see what and when changed in database schema. Dataedo 7 Enterprise has an option to enable tracking of schema changes.

Learn more

Search in HTML export

Now, it is able to search tables and other objects by their names, aliases and column names (and aliases) in static HTML exports.

Search in HTML

HTML export performance

We've completely redesigned our HTML export and now:

  • Loading time is much faster
  • HTML is more responsive
  • HTML is lighter
  • Time of export is much shorter
  • No need to choose between formats

Custom field types (drop-downs and more)

Until now, all custom fields were text based. Many users asked for an option to add a drop down or checkbox type custom field and we added it in this release.

From Dataedo 7.2 you can set a type for any custom field.

Available types are:

  • closed list with predefined values
  • open list with predefined values and ability to add new elements while editing
  • multiselect with predefined options
  • checkbox/binary
  • number (integer)
  • free text

Copy descriptions (beta feature)

Sometimes you have multiple databases with very similar structure and want to avoid documenting them multiple times, like Dev/UAT/Prod environments, or customizaions of databases for different customers. Now, you can have different documentations and copy descriptions between them to make the maintenance of separate documentations easier.

Copy documentation

Column view

Sometimes a column is used in a standard way in the entire database, and a specific description fits it multiple times.

Now, by right clicking a column you can find all columns with the same name in documentation or repository and edit multiple descriptions at the same time.

Column view opening

Column view

Improvements in data dictionary

User defined columns in existing tables

You can also add user defined columns to an existing table. Just select your table and click the Design table button in the ribbon, or right click the table you want to edit and choose Design table.

User defined columns in existing tables

Reordering columns in existing tables

In many DBMSes altering the order of columns in a table is not trivial. However logical column ordering can greatly improve understanding of their purpose.

From Dataedo 7.0.2 you can alter the order with which columns are displayed in your documentation.

Reordering columns in existing tables

Improved progress tracking

Many users requested a change in progress tracking to include only tables and columns. We added this as Tables & columns progress tracking option

Additionally, we've added an option to track progress of a specific custom field.

Description suggestions

Dataedo 7.2 now suggests texts for columns descriptions, titles or text custom fields based on texts from other columns. Those suggestions are hinted by blue marker and available in context menu.

Cross documentation modules

Previously, objects could only be added to modules within their documentation.

To better support cross database environments, you can now add object to modules from any documentation in your repository.

You can add objects from different documentations as usual, by dragging them from the navigation tree, from object lists one by one or in groups or by choosing modules on the object's description page.

Cross documentation module

Filtering and quick search

When your table contains hundreds of columns, finding the one you want to edit can be cumbersome. In Dataedo 7.1 we've added an option to filter objects:

Filter

And an addition to our previous search feature - search within current tab:

Quick search

This applies to almost all of our grid style tabs.

ERD improvements

We've made multiple improvements to our ERDs.

Add related tables

You can now add all objects related to a node on an ERD by right clicking it and choosing Add related tables:

Add related tables

Suggested entities

We redesigned and improved suggested entities list. It is now context aware and includes all available entities from repository and suggests entities assigned to module and related to selected entity/entities on the diagram.

Add related tables

Oracle improvements

Importing Packages

Dataedo 7.1 now imports Oracle packages:

Oracle packages

Importing Materialized Views

Dataedo 7 now imports materialized views from Oracle:

Materialized view

Direct connection

Until now, importing from Oracle required Oracle Client installation.

In Dataedo 7.1 you don't need to install Oracle Client to connect, you can now use direct connection option:

Oracle direct connection

Repository on Azure

Until now, Dataedo supported file and SQL Server based repository. In version 7 we added ability to set up Dataedo repository in Azure SQL Database.

As of release 7.0.3 beta Azure repository is still under development. It is stable, but requires manual setup.

Click here to find out more.

Azure repository

Much more to come...

Dataedo 6

Custom fields

This is a flagship feature of Dataedo 6 - the ability to add custom metadata fields to any database object/element. You can add additional descriptive fields to columns, tables, etc. This opens up a lot of new possibilities.

Those fields can be searched for, and used as filters during the search. Now you can find all confidential fields in your database easily.

Cross-database relationships and diagrams

Another key feature of 6.0 is the ability to create cross-database, cross-server, and cross-engine table relationships. It enables you to create relationships in Data Dictionary and use them to create ER diagram that shows multiple databases. This allows you to document how various systems are linked together. It's particularly useful in case of SQL Server and MySQL if your system has more databases as those engines don't support FKs across different databases.

One to one relations

Dataedo 6.1 introduced the ability to define cardinality of FK/relation to 1:1, both existing and user-defined relationships. In the data dictionary and ERDs.

User defined dependencies

You can now add a dependency to any object's dependencies tab. The new dependency can be created across databases and engines. This is particularly useful for databases such as MySQL where dependencies are not provided by database data dictionary, and you can document this information manually.

Just drag the objects you want to add the dependency on to either side of the dependencies tab.

User dependencies

Exporting descriptions back to the database

With Dataedo 6 you can save descriptions and custom fields from Dataedo repository to extended properties of your SQL Server databases or descriptions to comments of your Oracle databases.

New HTML layout and multiple databases

It is now possible to export multiple documentations to a single document. So if your system spans across multiple databases, you can hold them all in a single PDF file or a website.

We've altered the look of HTML export to make it even more intuitive.

Adding objects to ERD from navigation tree

Previously, to add an object to ERD you've had to use the object list on the right side of the screen. This could be cumbersome, especially when you've wanted to add objects from different documentations.

Now you can just drag any object you want to add from the navigation tree on the left side of the screen.

Drag objects from documentation tree

You can also drag entire branches to an ERD, eg. drag an entire module to the ERD to add all the tables and views included in it.

Drag objects from documentation tree

Adding objects from a different documentation to ERD

In Dataedo 6 it was possible to add objects from different documentaions to an ERD only if they had relations to existing objects.

Now you can add any object from any documentation to an ERD using the drag & drop feature described above.

Drag objects from documentation tree

Including multiple schemas in single Oracle import

Dataedo 6 can now import metadata from multiple Oracle schemas at once.

Scripts in exports

It is now possible to include view, function, stored procedure and trigger scripts in your documentation export.

You don't have to repeat information visible in script comments in descriptions anymore.

Improved FK View

Another small but pleasing change. We added table reference information next to a column. Now you can quickly see if a column is a foreign and to what table. This information is visible in UI and exported to HTML and PDF.

Improved manual keys view

From now on, all information from Keys tab (either imported from a database schema or added manually in Dataedo repository) will propagate to columns table, diagrams and exports.

Easier publishing on Confluence

We've added a special .js file to our HTML export making it even easier to include it on Confluence pages without issues with page sizing and formatting.

Easier to show/hide all columns on ER diagrams

We added a global option to show/hide columns on ER diagrams. You can now show all columns in all tables, hide all columns in all tables leaving only table header or show key columns.

SQL Server 2017 support

Dataedo 6.0 now officially supports new SQL Server 2017. Older versions should also work but were not tested.

Azure SQL Data Warehouse support

We've added support for Azure SQL Data Warehouse.

Found issue with this article? Comment below
0
There are no comments. Click here to write the first comment.