Dave Morris' Blog

Strange Behaviour of BizTalk 2004 XPath Expressions

2005-03-08T09:55:00Z

I've just been working on a problem with an XPath expression in an orchestration and thought its strangeness warranted a Blog - just to see if anyone else had come across it and to make other people aware of it really.

What I am trying to do is check the count of a certain child element in a message for zero and not to perform a send operation if it is zero. Sounds simple and here is the Boolean expression from my decide shape:

xpath(msg, "count(/*[local-name()='root']/*[local-name()='child'])")) != "0"

(Note I've abbreviated the XPath here before anyone says it's wrong - it is definitely right.)

That expression always evaluated to true. I even dumped out the result of the XPath as debug information to check and it was 0.

Confused, I changed the expression as follows:

System.Convert.ToInt32(xpath(msg, "count(/*[local-name()='root']/*[local-name()='child'])"))) != 0

Basically convert the XPath result to an integer and check its value against numeric zero rather than a string of zero. This works perfectly.

If anyone can explain this strange behaviour I'd appreciate it. If not at least its one for everyone else to be aware of.

Oh and I do have SP1 installed before anyone asks.

Scalar User Defined Functions and Computed Columns

2005-03-07T15:40:00Z

Came across an interesting SQL problem where a customer has a table with a primary key and X numerical fields. For each record the maximum value in these X fields was needed.

The issue was how to accomplish this efficiently? Or more accurately the least inefficient way of doing it is.

One option suggested was to have to UNION the table with itself X times and then do a MAX aggregation since there was a lot of data.

The more interesting option, which it turns out not many people are aware of, is to use a Scalar User Defined Function (UDF) to calculate the maximum value.

Here's an example of this in action:

CREATE FUNCTION CubicVolume
-- Input dimensions in centimeters
   (@CubeLength decimal(4,1), @CubeWidth decimal(4,1),
    @CubeHeight decimal(4,1) )
RETURNS decimal(12,3) -- Cubic Centimeters.
AS
BEGIN
   RETURN ( @CubeLength * @CubeWidth * @CubeHeight )
END

CREATE TABLE Bricks
   (
    BrickPartNmbr   int PRIMARY KEY,
    BrickColor      nchar(20),
    BrickHeight     decimal(4,1),
    BrickLength     decimal(4,1),
    BrickWidth      decimal(4,1),
    BrickVolume AS
              (
               dbo.CubicVolume(BrickHeight,
                         BrickLength, BrickWidth)
              )
   )

Basically the scalar UDF is called for each row returned from a SELECT. The column itself is not actually stored in the SQL Server table. Any INSERT and UPDATE on the table do not calculate the value for the computed column. Now obviously this is an overhead that may not be acceptable if the table is very static and read a lot. Here, it may be more efficient to perform the calculation once on INSERT or UPDATE of the table with INSTEAD OF triggers or another similar method.

WFA and BizTalk 2004

2005-03-04T15:03:00Z

It's well documented by many observers (and readily admitted by Microsoft) that Human Workflow Services (HWS) within BizTalk 2004 is not really up to the job for anything other than the most simple Workflow Automation (WFA) scenarios.

Looking at the roadmap for the Microsoft product set going forward:

There is nothing new for HWS in BizTalk 2006.
Then there is the muted introduction of a common orchestration platform (WinOE) for all Microsoft product beyond that.

It's easy to get confused about what is the right choice for a WFA solution on a Microsoft platform.

Once the scenario gets beyond HWS then this does not mean that BizTalk is not the right product for the WFA solution. In fact it will always likely be the most appropriate choice as it is increasingly rare for any WFA solution not to include a good deal or EAI too. See my previous post on BPM for a more in depth discussion of these topics.

So given all the confusion what is the best route forward? Adoption of a workflow product to sit alongside and closely integrate with BizTalk is the route. There are several product choices for this but the main players are:

1) K2.Net (www.k2workflow.com)
2) Teamplate (www.captaris.com)
3) Ultimus (www.ultimus.com)

The one I am going to consider in more depth is K2 - to start with all I'm going to highlight is some of the key Microsoft specific points around K2:

There is very close coupling with the whole Microsoft product suite - BizTalk obviously Active Directory, Office (including) InfoPath, SharePoint, Exchange, Outlook, Content Management Server, and even MSN Messenger.
The BizTalk integration is very close. K2 even uses BizTalk's Business Rules Engine for it's own rules management. It leverages BizTalk's BAM. K2 workflows can trigger and be triggered by BizTalk orchestrations.
The development environment integrates with Visual Studio.NET (supporting both C# and VB.NET) and ASP.NET.
A lot of the code needed is generated code as .NET code. This and allows things like integration of an InfoPath form or SharePoint site into a workflow calling BizTalk orchestrations - all without writing any code

Obviously like all WFA solutions K2 provides:

Management of assignment of tasks to people
Exceptions processing
Escalating of tasks based on business rules and / or schedule
Managing task queues for individuals and groups
Load balancing and optimizing task distribution
Service level management.

K2 also offers monitoring, tracking, and auditing of workflows and reporting on performance and quality of work.

Going forward in coming posts I want to look at some real life scenarios and development / deployment areas around solutions using K2.

BPM - EAI and WFA

2005-02-28T17:14:00Z

The whole Business Process Management (BPM) is a confusing space with many different packages performing many different aspects of a typical solutions requirements.

It is easy to split the BPM space into 2 distinct categories, each of which is converging:

Firstly we have Enterprise Application Integration (EAI) which takes an application centric (system-to-system) view of the world. Secondly there is Workflow Automation (WFA) which takes a people centric (people-to-people) view of the world. If we take any typical products in each of these spaces then there is a massive overlap between the two. And further, a typical BPM solution will encompass both EAI and WFA.

On the surface both a very similar, each providing the automation and management of business processes. Given that typically people will lump EAI and WFA products together into the single umbrella category of BPM. Taking this high-level view can cause a great deal of confusion and often lead to poor implementation choices.

As soon as we start to analyse things closer, we start to see significant differences between the requirements for EAI and WFA:

Speed: An EAI system has to perform tasks at machine speed since it caters for system-to-system processes. When moving information between systems or applications, speed and high volume processing are often a key requirement. Once we start dealing with the people centric world of WFA, speed remains important but is no longer a key consideration. With a workflow environment, transfer of tasks from one person to another could take many minutes or hours so the fact the WFA environments does not perform it in a fraction of a second is not an issue.
Number of Participants: There is always a limited number of enterprise applications that need to be involved in a process, this is typically around the half dozen make unless something is massively disfunctional within an organisation. An EAI solution only has to deal with this limited number of participants, each of which performs static operations - by this we mean that the roles these applications play are well defined and change infrequently. A WFA environment however may need to cater for a large number of participants both within and outside the organisation. A typical WFA solution can include anywhere between 100 and 2000 users. These users have different roles and responsibilities, all of which are in a constant state of change.
Handling Exceptions: WFA solutions have to handle large numbers of exceptions - people take time off, get reassigned, want to discuss tasks with someone else before performing them, or may want to delegate a task to someone else. The WFA environment must be able to cope with all of these and any other exception conditions that may occur. An EAI environment though does not deal with many exceptions because the application centric environment does not pose the problems of a people centric world. In an EAI solution, exceptions are driven by system and data failures and are relatively easy to trap and handle.
Business Rules: Within an EAI environment, business rules are data driven and can typically be handled programmatically. These data driven rules have the addtion of relationship based rules layered on top of them for a WFA solution. These are inherently more complex (especially as the size of the organisation and complexity of the relationships grows).
User Interface: An EAI solution has no user interface requirement since it is responsible for the exchange of data between enterprise applications. In contrast, user interface is the most complex requirements of WFA - it must provide a flexible user interface with dynamic forms and document management.
Data Transformation and Mapping: This is key to EAI solutions and the requirement for sophisticated data transformation and mapping capabilities is paramount. Each application within the enterprise typically saves data in its own format, thus the EAI environment must be capable of mapping information from one application to another. WFA has only limited data transformation and mapping requirements since it deals primarily with people centric situations.

Given all of this, it becomes clear why two distinct categories exist and typically products are optimised to perform one or the other.

BPM is converging though

While requirements for EAI and WFA are different, market requirements have produced an encompassing category - Business Process Management. So what is driving this convergence?

Business Processes are heterogeneous

More and more EAI processes are involving humans - whether to handle exceptions or provide human knowledge / judgement, people are being used in a optimised way. Similarly, many people centric solutions are being extended to incorporate enterprise applications to allow processes to be streamlined.
Web Services make integration simpler

Although there will always be a need for specialist, high-performance integration environments that EAI will cater for, the introduction of web services has lowered this barrier significantly. The use of a standard mechanism for performing typical integration tasks has dramatically reduced the cost of integration.
BPM must span multiple Enterprise Applications

Many enterprise applications provide embedded workflow management within the ERP or CRM environment. However, this route was limited since typically there are multiple enterprise applications and business processes span across these. The requirement for an external BPM / workflow environment was necessary.
Organisations are building BPM platforms

Becuase of the drivers for BPM to span the enterprise, many organisations are building a platform that includes EAI and WFA capabilities. This platform typically includes all the server components and design tools for building any BPM applications, and also the tools to operationally manage the lifecycle of business processes. These BPM platforms typically include the following:

Business Process Tools
Resource Management Tools
IT Developer Tools
Collaboration Tools
Repository
Administration Tools
Business Activity Monitoring
Workflow Engine
EAI Engine
Clients / User Interface / Forms
Web Services

Using BizTalk Pipelines to Split Records

2005-02-21T16:24:00Z

I've been looking into issues caused by using BizTalk disassembler pipeline components to split out records in a large flat file for processing in a project I am currently working on and have seen some interesting issues with it.

Technically this is a very easy thing to do and both the XML and Flat File disassembler components can be easily used to publish multiple messages into the Message Box from a single input document. There is a very good discussion of how this is achieved technically on Scott Woodgate's Blog. This is a very easy and consequently very tempting way of processing through records from an input file. For example, each line of the file is published as a separate message and processed individually by its own orchestration instance.

So why Blog about this then? Well what I'm really interested in here is the gotchas around doing this, particularly once the inputs start to get relatively big. All worth thinking about, but not obvious when you initially look at using this sort of processing.

On the project I am working on, the input file contains over 300,000 records and each is published as a seperate message and each is processed by a separate orchestration. This particular piece of processing was implemented on a previous phase of the project and has suddenly become an issue, although the volumes involved have not actually changed.

Where the issues occur are at the SQL Server level and concurrency with other processing at the same time - particularly that requiring relatively low latency. When the file is picked up, it is processed into the Message Box as a single transaction - rightly so since we do not want some of the records being published and others not. Obviously this is a very large transaction and uses up a lot of SQL Server resources while it is happening. The TLOG grows significantly and there is blocking of other processing occurring too.

Looking at the application server running the pipeline, it also saw a lot of resource issues, with it CPU pegged at 100% for some time (over an hour).

Both these resource issues cause other processing to slow down and even fail. Interestingly, once all the records are published, the 300,000 odd orchestrations process through without any real issue.

There are plenty of patterns around for how to process high-volumes in BizTalk so I am not going to go on about it here. For those who are interested, we now BCP the file into a database table and work through those in "manageable" batches using BizTalk and the SQL adapter. Both the application and database servers are now very happy!

BizTalk Time-Based Batching

2005-02-16T15:04:00Z

Think I'm going to start to get a reputation for going on about the same sort of thing all the time - hey it's been the bugging me for weeks so I'm going to tell everyone all about it.

There is a scenario I've been working on recently where IDOC's generated by an SAP system need to be batch together to form a single output flat file for an AS400 system. If you want more information about integrating BizTalk 2004 with SAP, see my colleague Tamer Shaaban's blog on the subject.

The issue here was how multiple IDOC's from SAP were correlated. The correlation was simply IDOC’s of a particular type within a given time window, e.g. all “delivery” IDOC’s within a 10 minute window of the first one received should go in the same batch file.

Easy really, correlate on the message type and use a sequential convoy to group them, in a loop until the time window completes. This is ok and works most of the time but has a couple of issues that would make it hard to support:

1) If a single IDOC processing fails for some reason, the whole batch processing fails.

2) There is always a processing window (albeit small) where messages can be discarded (zombies).

The latter of these isn’t that obvious but while the orchestration instance is running, all IDOC’s of that type will be correlated to that instance, even once it has finished receiving and is heading towards its exit point. The issue is you cannot “turn-off” a correlation set once you have received all messages you need into an instance of an orchestration. Normally this would not be an issue as the number of correlated messages being received is a determinate number. However, in this time-based scenario, the number of messages is totally indeterminate and there is no guarantee that after the last receive shape completes more messages won’t arrive and be correlated to the orchestration instance.

The solution to this to make the batch size determinate so that correlation is on a know number of messages and there cannot ever be zombies created. Also by splitting the processing into 2 phases (a pre-process and a batch-process), processing failures for individual IDOC’s can be isolated from the batch.

The following diagram shows the basic pattern for this processing:

The key to this is the “repository” in the middle. This is there purely for assigning each message received into a batch and determining when a batch completes (and consequently its size). By using this mechanism we are now determinate in our batch processing and can guarantee no zombies.

Zombies and Convoys

2005-02-16T14:22:00Z

Just so you know, because I wouldn't want to lead anyone on, I’m talking BizTalk 2004 here and not the somewhat dodgy 1980’s horror film by Jesus Franco, Oasis of the Zombies.

I’ve recently spent some time looking into a Singleton pattern for orchestrations to apply to the current project I am working on. Basically we have a lot of interfaces that are scheduled in particular time slots but can only run once and must act as singletons.

This sounds initially very easy to do with a sequential convoy but is prone to zombies – i.e. the orchestration completes with discarded messages. This occurs when messages that are part of the convoy set are queued for the orchestration but not consumed by it.

The topic of zombies itself is much wider than this narrow scenario I am looking at here but there are some very good details on the subject in the BizTalk Core Engine’s WebLog.

Since the length of processing time for any particular singleton is indeterminate the challenge has been to make sure all messages are consumed in every case. Take the following example:

An interface is “kicked off” by its schedule every 15 minutes and its typical processing is within one minute so it’s never likely to receive multiple “kick off” messages in a single instance. However, at month end, the volume is much higher and it can take as much as 4 hours to process, maybe a bit longer, maybe a bit shorter. Here all subsequent “kick off” messages need to be consumed while it is running to stop multiple instances from running.

Here’s the basic solution that uses a parallel action in its processing, one branch consuming convoy messages and the other performing the singleton’s processing.

It looks pretty complicated but is actually very straightforward and involves synchronised scopes and a completed flag. The processing branch sets a flag to say it is finished and the next time the receive branch times out it spots this and breaks out of its loop.

Why synchronised scopes? Basically this is needed to allow parallel branches to update the same variables without trampling all over each other - it makes the orchestration thread safe. The BizTalk development environment is good enough to not allow you to be unsafe and if you try and use the same variable in multiple branches you will get a build error. Essentially it serialises access to the data.

So why shout about something so simple? Well there are a few gotchas here.

First of all by only checking for completion in a timeout situation, you are guaranteeing not to leave zombies if there are several queued up – obviously your timeout needs to be much shorted than the scheduling frequency. Here I’ve got a 10 second timeout with a likely scheduling frequency of around 15 minutes. This is a race condition but by being sensible about the timeout versus schedule frequency, it should never have an effect on the processing.

Next is the use of two variables, one for the loop condition and one for the finished state. This is so the size of the synchronised scope in the receive branch can be as small as possible to limit the locking of the variables. If only one variable is used, this means that a scope is needed around the entire loop. The problem this causes is that is locks the loop condition variable so the processing branch cannot set it. Essentially everything deadlocks.

Also be aware of the scopes themselves. Here I’ve only got 2, one for checking the finished state and one for processing. Both are non-transactional. However, if they are transactional, then their locking of variables is moved out to the scope level rather than the individual shape level and again we can block. The checking blocks until the main processing scope completes. It then immediately sees the finished state set true and falls out the loop. Since its been blocking on the check, there may well be queued messages for the orchestration that it has not consumed – zombies again. I have a scenario where the main processing is in a transaction where the setting of the finished flag has been moved out into its own non-transactional scope after the main processing scope.

The last issue I saw was quite unexpected and occurred only because I was being overly anal about my testing for some reason – must have been bored! If you leave the orchestration stopped but enlisted and queue up messages for it everything looks just fine. However once you start up, the instance always completes with discarded messages. The first message submitted is always received twice – once to activate the instance and then again as the first correlated message.

After a bit of head scratching it was straight onto Microsoft support since this is a blatant bug, although one we are unlikely to hit in this particular scenario. It is the case for any sequential convoy though. Here all subsequent messages are just swallowed to prevent further processing but imagine if you were actually processing them and adding up data! There will be a fix available for this soon - once it has been verified. Be aware it has not made it into SP1 though. (See my colleague Matt Hall’s Blog on the Contents of SP1.)

I’ve packaged up the sample solution that can be downloaded from here.