Welcome to EMC Consulting Blogs Sign in | Join | Help

SSIS Junkie

SSIS: Caching result sets for re-use in a ForEach loop

Peter (sorry, don't know your surname) made a very good comment here: http://blogs.conchango.com/jamiethomson/archive/2006/02/20/2895.aspx#2903 that data-flows in a ForEach loop carry out repeated SELECTs against a data source and it would be good to be able to cache that data rather than executing the same query over and over.

Its a great point and one that I wholeheartedly endorse. Hopefully Peter has raised it at the Feedback Centre. In the meantime, there is a workaround, of sorts, that can be used in certain situations.

Peter's example talked of an OLE DB Source component that was being used in a MERGE JOIN component. He reasoned that seeing as that data wasn't changing for each iteration then the data could be cached somewhere. Well, how about populating a raw file with that data prior to the ForEach loop? That way the data source is only accessed once and the raw file can be used repeatedly as the source for the MERGE JOIN. Sure, its not quite as ideal as caching the data in memory but raw files are supremely fast and there is virtually no overhead in using them so speed should not be a problem. The main benefit here is of course that you don't put undue load on your RDBMS but also, it doesn't tie up memory resource.

If the data cache is absolutely required to be held in memory then using an ADO recordset to store the data (as explained here: http://blogs.conchango.com/jamiethomson/archive/2006/01/04/2540.aspx) could be done instead.

I find it useful to remember that there is usually more than one way to achieve a desired action in SSIS. Explore the options that are available and you may find that better options exist that the obvious one - as exhibited here!

-Jamie

 

 

Published 20 February 2006 17:08 by jamie.thomson

Comments

 

Peter said:

I have to say that this response was pretty fast. I didn't even expect to get a response - just posting on something I'd experienced. My actual need for this job has passed, but I'll remember it in the future. The ADO recordset doesn't seem as if it would be practical as we're looking at abou 240k rows. The flat file could work really well, though. I seem to remember this was just a "short-term" solution for a small number of columns used to populate some additional information and eventually pump the data into an excel sheet for someone to look at.

I'll flag this as something to try next time. I think this was entered in the feedback center, but I've lost track of what I added regarding SSIS.

-Pete (Schott)
February 20, 2006 23:49
New Comments to this post are disabled

This Blog

Syndication

Powered by Community Server (Personal Edition), by Telligent Systems