blogs & Things

How To Inspect Items When Using The Execute Pipeline Activity In ADF/Synapse Pipelines

blogs & Things
wooden toy

Greg Roberts – cloudThing, Data Scientist

A workaround for ADF/Synapse pipelines when using the Execute Pipeline activity to inspect anything about the run from the calling pipeline

 

A longstanding issue I’ve found with ADF/Synapse pipelines is that if you try to use the Execute Pipeline activity, then it won’t really let you inspect anything about the run from the calling pipeline.

 

This can be generalised to state that within any individual pipeline, you’ll never really be able to understand much about other pipeline runs.

There’s actually a big Azure Feedback post about this, but unfortunately no one seems to have come up with any kind of workaround… So, here’s my workaround…

 

Just call the Synapse/Data Factory API from within the pipeline!

Setup the Rest API as a LinkedService:

 

 

  • Auth type = Managed Identity. It’s worth noting that AAD Resource is not the usual endpoint used for managed identity authentication (see this note)
  • For the base URL, just have workspace_name as a parameter for the linked service. This will be used later and will make this linked service agnostic to the environment you’re in.

 

Then create a dataset from this:

 

 

As you can tell, I’m  being really generic here and just passing the relative url through for now.

Again, it’s worth noting that workspace_name is a parameter here… we only define it in the actual pipeline.

 

Et voila!

Now we have a dataset we can use to get data from the Data Factory API. e.g.:

 

 

As you can now see, we can pass in the workspace name, so this linked service can be moved to new environments with no issue!

 

I actual had to do a double take when I first hit preview data…

 

 

We can now easily get the top level output of a run.

 

“But what about details of the inner activities” you ask…?

Glad you did! Just use this method and hey presto, we now have all the details of our pipeline run…

 

 

 

 

Since this is a REST dataset, we unfortunately can’t use it immediately in a Lookup or Get Metadata activity… but, we can easily persist this to a storage account, then look it up and understand anything about how our pipeline ran, from within a pipeline!

 

This was fun little workaround to create and saved me from a jam.

As I couldn’t find anyone else talking about having done this, I thought I’d leave it here in case you found it useful.

More blogs & Things

More blogs & Things


James Crossland in NonProfit

AI + Automation: Reducing Donor Churn & Maintaining Sponsor Interest

Churn management is a vital element of any marketing strategy, and the NonProfit sector is no exception. Knowing what to track and having a joined up view of all your donations data is vital for getting this right, and also opens the door to building innovative data-driven campaigns.   At our recent DataScience and Transformation in Charities […]


James Crossland in NonProfit

Dynamics 365 In NonProfit’s

Charities have unique funding concerns, and an obligation to spend as much as possible on their chosen cause. However, an investment in technology can offer ROI in the form of more than just improved fundraising. Dynamics 365 can help rework complex business processes, ensure compliance with stringent safeguarding and financial regulations, as well as consolidate […]


James Crossland in Tech

8 Ways Your Business Can Increase Turnover With Big Data

Understand how Big Data and Data Science can transform your business…   Big Data is the phrase that’s used to categorise any data that’s too large, complex, cumbersome or complicated to be managed and processed by conventional technology. To put that into a relatable context; being able to recommend your customers content, products or offers based […]


James Crossland in NonProfit

How To Reduce Donor Churn In NonProfits

Reducing Donor Churn doesn’t have to be a big task but does need to be a fundamental part of a NonProfit’s day to day processes   What Is Donor Churn? Donor Churn is the likelihood of an individual stopping their donations to a charitable cause for a variety of different reasons resulting in the non-profit organisation […]


James Crossland in Tech

Agile: Cutting Costs, Improving Quality & Accessing Talent

After using Agile to develop software products for several years, we thought we’d share the challenges we encountered at the start, what we did to change and the results we saw (which were ultimately uplifts in quality and efficiency)…   My development team has been using Agile to develop software product since 2007. Personally, I’ve seen many […]


James Crossland in Tech

UI VS UX

What’s the difference between UI and UX?   Simply put UI (or User Interface) are the pages, screens, buttons, icons and any other visual aspects of a website or App that let you interact with it… or to expand on that into the non-virtual world… UI is how you experience using something – For instance in opening a fridge, […]


Send us a Message






    Call cloudThing
    0121 393 4700
    DEVELOP • DYNAMICS • DEVOPS • DATA
    By pressing send you agree to our Terms & Conditions