Azure Synapse Pipelines is a cloud-based data integration and orchestration service provided by Microsoft Azure. It allows you to create, schedule, and manage data workflows, as well as move and transform data across various sources and destinations.
To obtain a complete lineage for your Azure Synapse Analytics environment, we highly recommend to import data into Dataedo using both the Azure Synapse Analytics connector and the Azure Synapse Pipelines connector.
Supported elements and metadata
- Pipelines as ETL Programs
- Folder (as schema)
- Activities as Processes in Data Lineage
- Data lineage (object and column level, details below)
- Integration Datasets as Datasets
- Folder (as schema)
Dataedo imports all the activities as data processes within pipelines. Nested activities like ForEach, Until, IfCondition, and Switch are also imported with all the activities inside them.
Activites we build automatic data-lineage for
- Copy activity - Object-level and column-level lineage
- Dataflow - Object-level lineage
Automatic data lineage
Dataedo Azure Synapse Pipelines connector always creates automatic data lineage dataset -> task -> dataset. We also support source -> dataset -> task -> dataset -> sink, but currently only for:
- Microsoft SQL Server
- Amazon Redshift
- Azure Synapse Analytics
Connecting to Azure Synapse
To list resource groups in a connection window you must belong to the Azure Synapse Contributor Role role at the Resource Group level or above.
To run import you need just read access to the workspace you want to document. However in this case you must provide the resource group and workspace name by hand instead of picking from the list.
Add new connection
To connect to Azure Synapse Pipelines and create new documentation click Add documentation and choose Database connection.
On the connection screen choose Azure Synapse Pipelines as DBMS.
- Subscription - Azure subscription assigned to your workspace.
- Resource group - Resource group where your workspace sits in.
- Workspace - Name of the workspace you want to extract metadata from.
Go to portal.azure.com . Search for your Azure Synapse Analytics workspace name and open this resource to get information needed.
When the connection was successful Dataedo will read objects and show a list of objects found. You can choose which objects to import. You can also use an advanced filter to narrow down the list of objects.
Confirm list of objects to import by clicking Next.
The next screen allows you to change the default name of the documentation under your schema will be visible in the Dataedo repository.
Click Import to start the import.
When done close import window with Finish button.
Your Azure Synapse Analytics objects has been imported to new documentation in the repository.