-
Notifications
You must be signed in to change notification settings - Fork 0
Workflow Components and Operation
The Workflows feature allows for ETLs to be automatically triggered with dependency on multiple combinations of data sources surpassing predefined quality score thresholds.
The bulleted numbers below correspond with the numbered callouts in the below screenshot.
The Schedule Run Time is used in conjunction with the Alerts. Alerts can be set to send specified users an email notification if the last ETL in a Workflow doesn’t trigger by the designated Scheduled Run Time. In the above screenshot, the last ETL in the example Workflow would be ‘AmultiPerfPipe_copy5’.
When creating Workflows, you must select Data Sources that are already set up in the platform. These Data Sources will also be found in the Sources tab in the blue left-hand navigation pane. The highlighted portion in the above screenshot is the parameter that is used when setting up an ETL so the ETL knows which file to process when triggered.
- If you click on the parameter name which is highlighted, it will automatically save to your clipboard.
- This parameter name can also be found in the Settings tab in the Manage screen for any Source.
The Run ETL button will manually trigger the designated ETL. In our above instance, this would trigger the ETL circled in #5.
When a DataSource has a report run, the status will display as “Complete” in green (as shown in the screenshot). The Data Quality Score for that report run will also display. The ‘Min DQ Score’ is a threshold that a source’s quality score must surpass in order to auto-trigger the ETL. In the above example, the report score did not surpass the threshold which is indicated by the Score being highlighted in red.
- If a DataSource is used multiple times in either the same Workflow or in different Workflows, the Min DQ score will be the same for every instance where that DataSource is present within all Workflows. In other words, if the same DataSource is found in two different Workflows, and you change the Min DQ Score on one of them, it will automatically update the other with the same Min DQ Score.
For information purposes only.