Welcome to EMC Consulting Blogs Sign in | Join | Help

Paul Petrov's blog

  • Microsoft BizTalk 2010 - Line of Business Systems Integration

    I just finished reading new book on Microsoft BizTalk 2010 written by BizTalk colleagues Kent Weare, Richard Seroter, Thiago Almeida, Sergei Moukhnitski, and Carl Darski. There are many good books that explore core BizTalk features for beginners, intermediate, and advanced developers. This one stands out by the fact that it covers relatively less documented aspect of BizTalk development – integrating with different line of business applications. The subject is vast and diverse so attempt to fit it in one book is a challenge by itself.

    The book opens with the chapter on WCF Adapters SDK which can be good introduction into the world of WCF based adapters. It explains high level design and idea of LOB adapters and WCF bindings. The chapter contains a simple application example that uses custom WCF adapter. While working through this exercise readers will get acquainted with adapter metadata, binding configuration, endpoint URI, SOAP action mapping, and other fundamental concepts of the WCF enabled adapters.

    Next chapter explores probably the most frequently used and well-known of the WCF Adapters – WCF SQL Adapter. All features of this adapter including more advanced like typed polling, notification and debatching are covered in details with examples. It makes this chapter great practical WCF-SQL handbook for all levels from beginner to advanced user and an excellent addition to the MSDN documentation.

    Then follow few chapters each dedicated to one specific LOB application integration: Microsoft Dynamics CRM, WCF SAP Adapter, SharePoint, Dynamics AX, and SalesForce.  Every chapter has overview of the Line of Business application, its role in business processes, and then goes into detailed integration example with explanations, sample code, and screen shots.

    Separate consideration deserves chapter on integration with Windows Azure Platform AppFabric. This is a new exciting feature of the BizTalk 2010 that expands its capabilities into the cloud based service bus solutions. This chapter alone makes the book worth having.

    Overall, authors and editors has done a great job covering such wide and important area of BizTalk functionality. Sure, it’s impossible to go into very deep details and explore all intricacies of every LOB application in just one book. Yet this one is a very good starting point for any consultant in the field who has to deal with disparate systems integration on a daily basis.

  • BizTalk 2010 Certification Exam

    I took a shot at new (to me) certification exam for BizTalk 2010. I was able to pass it without any preparation just based on the experience. That does not mean this exam is a very simple one. Comparing to previous (2006 R2) it covers some new areas (like WCF) and has some demanding questions and situation to think about. But the most challenging factor is broad feature coverage. Overall, the impression that if BizTalk continues to grow in scope it’s better to create separate exams for core functionality and extended features (like EDI, RFID, LOB adapters) because it’s really hard to cover vast array of BizTalk capabilities.

    As far as required knowledge and questions allocation I think Microsoft description is on target. There were definitely more questions on deployment, configuration and administration aspects comparing to previous exam. WCF and WCF based adapters now play big role and this topic was covered well too. Extended functionality is claimed at 13% of the exam, I felt there were plenty of RFID questions but not many EDI, that’s why I thought it’d be useful to split exam into two to cover all of them equally. BRE is still there and good, cause it’s usually not very known/loved feature of the package.

    At the and, for those who plan to get certified, my advice would be to know all those areas of BizTalk for guaranteed passing: messaging and orchestrations, core adapters, routing, patterns; development of all artifacts and orchestrations; debugging and exceptions handling; packaging, deployment, tracking and administration; WCF bindings and adapters; BAM, BRE, RFID, EDI, etc. You may get by not knowing one smaller non-essential part (like I did with RFID, for example). In such case you better know all other areas very well to cover for the weak spot. If there more than one whiteouts in the knowledge it’s good idea to study and prepare: MSDN, blogs, virtual labs and good VM to play with can help when experience is not enough. So best wishes and good skill to you in passing this certification!

  • ESB Off-Ramp with Dynamic Resolver to WCF-SQL Binding

    This example is not covered in the ESB Toolkit samples and I bumped my forehead a few times while making it work. I thought it’d be helpful to save other fellow BizTalk'ers from headaches by publishing findings.

    I have been fun of dynamic generic messaging for quite a while and went long ways to avoid working with typed messages and static bindings when it made sense. Nowadays, with the ESB Toolkit one does not have to spend much effort to achieve this goal. But one has to learn intricacies of configuration to unleash unlimited flexibility of the Toolkit.

    Let's say we have a requirement to store some messages on service bus to the database. And the number of message types supported is growing plus their schemas can potentially change. If the database is SQL Server we can use an ESB Off-Ramp dynamically bound to WCF-SQL send port.

    The send port interface can be implemented different ways. One way is to generate typed WCF-SQL Adapter schemas for database objects (tables or procedures). Then create maps to transform original message to the adapter schema. This may potentially lead to complex maps that perform poorly, difficult to maintain and support. We also have to deploy new schemas every time new type is added.

    Other way with minimal BizTalk coding is to use one-way port calling stored procedure that accepts entire message as a parameter and internally maps/saves message to the database. This saves us from creating and deploying stored procedure schema and map, as well as schema for the response message. We also gain performance on the BizTalk end by offloading XML to relational model mapping to the database engine which it does more efficiently. Simply put, stored procedures perform better and simpler to maintain. I chose this approach over the first.

    Suppose, we have two message types http://contoso.com/schemas#PurchaseOrder and http://contoso.com/schemas#Invoice. We have created two stored procedures InsertPurchaseOrder and InsertInvoice. Each stored procedure accepts one parameter, the message body:

    CREATE PROCEDURE InsertPurchaseOrder 
        @messageBody nvarchar(max)
    AS
    BEGIN
        SET NOCOUNT ON;
        
        DECLARE @docHandle int
        EXEC sp_xml_preparedocument @docHandle OUTPUT, @messageBody, N'<ns0:PurchaseOrder xmlns:ns0="http://contoso.com/schemas" />'
    
    -- Perform mapping and insert here using OPENXML for example
    
    END
    

    Create BRE policy to resolve endpoint configuration:

    In the Action area there are a lot of things being set up. Let's go through them one by one:

    Parameter

    Value

    Notes

    Set Endpoint Message Exchange Pattern

    One-Way

    Establishes exchange pattern, we don’t need solicit-response in our case

    Set End Point Outbound Transport Type

    WCF-Custom

    This is the binding we are going to use.

    Note: this value is not available in the ESB.TransportTypes vocabulary. Thus, we use literal string value “WCF-Custom”

    Set Endpoint Outbound Transport Location

    mssql://sqlservername//databasename?

    This is standard form of the WCF-SQL uri. The example given is for default SQL instance (notice double slash)

    Set End Point WCF Action

    {Procedure/dbo/InsertPurchaseOrder}

    This is from WCF-SQL adapter schema, but the trick here, it wants to be enclosed in {} which is not obvious

    Set End Point Target Namespace

    http://schemas.microsoft.com/Sql/2008/05

     

    Set End Point Config

    See below

    That’s where “magic” is configured and deserves separate consideration

    Endpoint configuration details

    BindingType=sqlBinding&OutboundXmlTemplate=<InsertPurchaseOrder xmlns="http://schemas.microsoft.com/Sql/2008/05/Procedures/dbo"><messageBody><bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007" encoding="string"/></messageBody></InsertPurchaseOrder>&OutboundBodyLocation=UseTemplate&BindingConfiguration=<binding name="sqlBinding" />

    Parameter

    Value

    Notes

    BindingType

    sqlBinding

     

    BindingConfiguration

    <binding name="sqlBinding" />

    Default binding configuration which can be augmented.

    OutboundXmlTemplate

    <InsertPurchaseOrder xmlns="http://schemas.microsoft.com/Sql/2008/05/Procedures/dbo">

    <messageBody>

    <bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007" encoding="string"/>

    </messageBody>

    </InsertPurchaseOrder>

    This is key parameter to make adapter accept the message and pass to stored procedure. The root element is the procedure name. The immediate child element is the procedure’s parameter. The enclosed element is to tell where to place message body.

    OutboundBodyLocation

    UseTemplate

    Tells adapter to use template rather than raw message body.

    This is another parameter which was not apparent since there’s no enumeration to provide possible values. Digging into adapter reference helps.

    Add Messaging service to the Itinerary with the BRE endpoint resolver that uses policy created earlier. Then simply use one-way off-ramp. In the dynamic one-way port configuration PassThroughTransmit pipeline will do the job.

    Itinerary

    Once all these components implemented you have truly generic infrastructure to save any message to the database with minimal overhead. As new messages added, you simply create new stored procedure and rule that associate message type with it. And there are no BizTalk artifacts to deploy.

  • Configure MySQL .NET Connector with Entity Framework

    I've been working lately on web application that uses Asp.Net MVC, WordPress and MySQL database. To communicate from .NET code to MySQL database using Entity Framework I installed MySql .NET Connector. It worked fine out of the box on developer's workstations but once deployed to hosted web server application started to throw following error:

    [NotSupportedException: Unable to determine the provider name for connection of type 'MySql.Data.MySqlClient.MySqlConnection'.]
    System.Data.Entity.ModelConfiguration.Utilities.DbConnectionExtensions.GetProviderInvariantName(DbConnection connection)
    System.Data.Entity.ModelConfiguration.Utilities.DbConnectionExtensions.GetProviderInfo(DbConnection connection, DbProviderManifest& providerManifest)
    System.Data.Entity.DbModelBuilder.Build(DbConnection providerConnection)
    System.Data.Entity.Internal.LazyInternalContext.CreateModel(LazyInternalContext internalContext)
    System.Data.Entity.Internal.RetryLazy`2.GetValue(TInput input)
    System.Data.Entity.Internal.LazyInternalContext.InitializeContext()
    System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType)
    System.Data.Entity.Internal.Linq.InternalSet`1.Initialize()
    System.Data.Entity.Internal.Linq.InternalSet`1.get_InternalContext()
    System.Data.Entity.Internal.Linq.InternalSet`1.ActOnSet(Action action, EntityState newState, Object entity, String methodName)
    System.Data.Entity.Internal.Linq.InternalSet`1.Add(Object entity)
    System.Data.Entity.DbSet`1.Add(TEntity entity)

    Apparently, somehow MySQL .NET provider was not registered on server although installation was done properly and all assemblies were present in GAC. Since on shared hosting I did not have access to machine.config by adding the same registration entry to the application web.config file resolved the issue:

     <system.data> 
        <DbProviderFactories> 
          <add name="MySQL Data Provider" invariant="MySql.Data.MySqlClient" description=".Net Framework Data Provider for MySQL" type="MySql.Data.MySqlClient.MySqlClientFactory, MySql.Data, Version=6.4.3.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d"/> 
        </DbProviderFactories> 
      </system.data>

    Important:

    1) Make sure to register the same version that is installed on the server
    2) Provide precise fully qualified name in the type attribute. For example, adding extra space between MySql.Data.MySqlClient.MySqlClientFactory and MySql.Data will make this entry useless and will result in the same error

  • Hotfix for BizTalk WCF Receive Adapter Issue

    Microsoft has released official hotfix related to the problem I described earlier (see: http://geekswithblogs.net/paulp/archive/2010/05/17/139876.aspx). The knowledge base article and link to hotfix download are located at http://support.microsoft.com/kb/2300507/en-us

  • Integration with Multiple Versions of BizTalk HL7 Accelerator Schemas

    Microsoft BizTalk Accelerator for HL7 comes with multiple versions of the HL7 implementation. One of the typical integration tasks is to receive one format and transmit another. For example, system A works HL7 v2.4 messages, system B with v2.3, and system C with v2.2. The system A is exchanging messages with B and C. The logical solution is to create schemas in separate namespaces for each system and assign maps on send ports. Schematic diagram of the messaging solution is shown below:

     

    Nothing is complex about that conceptually. On the implementation level things can get nasty though because of the elaborate nature of HL7 schemas and sheer amount of message types involved. If trying to implement maps directly in BizTalk Map Editor one would quickly get buried by thousands of links between subfields of HL7 segments. Since task is repetitive because HL7 segments are reused between message types it’s natural to take advantage of such modular structure and reduce amount of work through reuse. Here’s where it makes sense to switch from visual map editor to good old plain XSLT. The implementation is done in three steps.

    First, create XSL templates to map from segments of one version to another. This can be done using BizTalk Map Editor subsequently copying and modifying generated XSL code to create one xsl:template per segment. Group all segments for format mapping in one XSL file (we call it SegmentTemplates.xsl). Here’s how template for the PID segment (Patient Identification) would look like:

      <xsl:template name="PID">
        <PID_PatientIdentification>
          <xsl:if test="PID_PatientIdentification/PID_1_SetIdPatientId">
            <PID_1_SetIdPid>
              <xsl:value-of select="PID_PatientIdentification/PID_1_SetIdPatientId/text()" />
            </PID_1_SetIdPid>
          </xsl:if>

          <xsl:for-each select="PID_PatientIdentification/PID_2_PatientIdExternalId">
            <PID_2_PatientId>
              <xsl:if test="CX_0_Id">
                <CX_0_Id>
                  <xsl:value-of select="CX_0_Id/text()" />
                </CX_0_Id>
              </xsl:if>
              <xsl:if test="CX_1_CheckDigit">
                <CX_1_CheckDigitSt>
                  <xsl:value-of select="CX_1_CheckDigit/text()" />
                </CX_1_CheckDigitSt>
              </xsl:if>
              <xsl:if test="CX_2_CodeIdentifyingTheCheckDigitSchemeEmployed">
                <CX_2_CodeIdentifyingTheCheckDigitSchemeEmployed>
                  <xsl:value-of select="CX_2_CodeIdentifyingTheCheckDigitSchemeEmployed/text()" />
                </CX_2_CodeIdentifyingTheCheckDigitSchemeEmployed>
              </xsl:if>

    . . . rest is omited for brevity

        </PID_PatientIdentification>
      </xsl:template>

    This is the most tedious and time consuming part. Templates can be created for only those segments that are used in message interchange. Once this is done the rest goes much easier.

    The next step is to create message type specific XSL that references (imports) segment templates XSL file. Inside this file simple call segment templates in appropriate places. For example, beginning of the mapping XSL for ADT_A01 message would look like code below. It simply calls segment template directly for required singular elements and in for-each loop for optional/repeating elements.

      <xsl:import href="SegmentTemplates_23_to_24.xslt" />
      <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />

      <xsl:template match="/">
        <xsl:apply-templates select="s0:ADT_A01_23_GLO_DEF" />
      </xsl:template>
      <xsl:template match="s0:ADT_A01_23_GLO_DEF">
        <ns0:ADT_A01_24_GLO_DEF>
          <xsl:call-template name="EVN" />
          <xsl:call-template name="PID" />
          <xsl:for-each select="PD1_PatientDemographic">
            <xsl:call-template name="PD1" />
          </xsl:for-each>
          <xsl:call-template name="PV1" />
          <xsl:for-each select="PV2_PatientVisitAdditionalInformation">
            <xsl:call-template name="PV2" />
          </xsl:for-each>

    And lastly, create BizTalk map (btm) that references message type specific XSL. It is essentially empty map with Custom XSL Path set to appropriate XSL:

     

    In the end, you will end up with one segment templates file that is referenced by many message type specific XSL files which in turn used by BizTalk maps. Once all segment maps are created they are widely reusable and all the rest work is very simple and clean.

  • Mapping Repeating Sequence Groups in BizTalk

    Repeating sequence groups can often be seen in real life XML documents. It happens when certain sequence of elements repeats in the instance document. Here's fairly abstract example of schema definition that contains sequence group:

    <xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003"

               xmlns:xs="http://www.w3.org/2001/XMLSchema"

               xmlns="NS-Schema1"

               targetNamespace="NS-Schema1" >

      <xs:element name="RepeatingSequenceGroups">

        <xs:complexType>

          <xs:sequence maxOccurs="1" minOccurs="0">

            <xs:sequence maxOccurs="unbounded">

              <xs:element name="A" type="xs:string" />

              <xs:element name="B" type="xs:string" />

              <xs:element name="C" type="xs:string" minOccurs="0" />

            </xs:sequence>

          </xs:sequence>

        </xs:complexType>

      </xs:element>

    </xs:schema>

    And here's corresponding XML instance document:

    <ns0:RepeatingSequenceGroups xmlns:ns0="NS-Schema1">

      <A>A1</A>

      <B>B1</B>

      <C>C1</C>

      <A>A2</A>

      <B>B2</B>

      <A>A3</A>

      <B>B3</B>

      <C>C3</C>

    </ns0:RepeatingSequenceGroups>

    As you can see elements A, B, and C are children of anonymous xs:sequence element which in turn can be repeated N times. Let's say we need do simple mapping to the schema with similar structure but with different element names:

    <ns0:Destination xmlns:ns0="NS-Schema2">

      <Alpha>A1</Alpha>

      <Beta>B1</Beta>

      <Gamma>C1</Gamma>

      <Alpha>A2</Alpha>

      <Beta>B2</Beta>

      <Gamma>C2</Gamma>

    </ns0:Destination>

    The basic map for such typical task would look pretty straightforward:

    If we test this map without any modification it will produce following result:

    <ns0:Destination xmlns:ns0="NS-Schema2">

      <Alpha>A1</Alpha>

      <Alpha>A2</Alpha>

      <Alpha>A3</Alpha>

      <Beta>B1</Beta>

      <Beta>B2</Beta>

      <Beta>B3</Beta>

      <Gamma>C1</Gamma>

      <Gamma>C3</Gamma>

    </ns0:Destination>

    The original order of the elements inside sequence is lost and that's not what we want. Default behavior of the BizTalk 2009 and 2010 Map Editor is to generate compatible map with older versions that did not have ability to preserve sequence order. To enable this feature simply open map file (*.btm) in text/xml editor and find attribute PreserveSequenceOrder of the root <mapsource> element. Set its value to Yes and re-test the map:

    <ns0:Destination xmlns:ns0="NS-Schema2">

      <Alpha>A1</Alpha>

      <Beta>B1</Beta>

      <Gamma>C1</Gamma>

      <Alpha>A2</Alpha>

      <Beta>B2</Beta>

      <Alpha>A3</Alpha>

      <Beta>B3</Beta>

      <Gamma>C3</Gamma>

    </ns0:Destination>

    The result is as expected – all corresponding elements are in the same order as in the source document. Under the hood it is achieved by using one common xsl:for-each statement that pulls all elements in original order (rather than using individual for-each statement per element name in default mode) and xsl:if statements to test current element in the loop:

      <xsl:template match="/s0:RepeatingSequenceGroups">

        <ns0:Destination>

          <xsl:for-each select="A|B|C">

            <xsl:if test="local-name()='A'">

              <Alpha>

                <xsl:value-of select="./text()" />

              </Alpha>

            </xsl:if>

            <xsl:if test="local-name()='B'">

              <Beta>

                <xsl:value-of select="./text()" />

              </Beta>

            </xsl:if>

            <xsl:if test="local-name()='C'">

              <Gamma>

                <xsl:value-of select="./text()" />

              </Gamma>

            </xsl:if>

          </xsl:for-each>

        </ns0:Destination>

      </xsl:template>

    BizTalk Map editor became smarter so learn and use this lesser known feature of XSLT 2.0 in your maps and XSL stylesheets.

  • Referring EDMX file in Separate VS Project from T4 Template

    In my project I needed to separate template generated entities, context in separate projects from the EDMX file. I’ve stumbled across this problem how to make template generator to find edmx file without hardcoding absolute path into the template. Using relative path directly (inputFile=@”..\ProjectFolder\DataModel.edmx”) generated error:

    Error      2              Running transformation: System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\ProjectFolder\DataModel.edmx'

    The code that worked well for me when placed in the beginning of the .tt file:

    string rootPath = Host.ResolvePath(String.Empty);

    string relativePath = @"..\\ProjectDir\\DataModel.edmx";

    string inputFile = Path.Combine(rootPath, relativePath);

    EdmItemCollection ItemCollection = loader.CreateEdmItemCollection(inputFile);

  • BizTalk HL7 Receive Pipeline Exception

    If you experience sequence of errors below with BizTalk HL7 MLLP receive ports you may need to request a hotfix from Microsoft. Knowledge base article number is 2454887 but it's still not available on the KB site. The hotfix is recently released and you may need to open support ticket to get to it. It requires three other hotfixes installed:
    • 970492 (DASM 3.7.502.2)
    • 973909 (additional ACK codes)
    • 981442 (Microsoft.solutions.btahl7.mllp.dll 3.7.509.2)

    If the exceptions below repeatedly appear in the event log you most likely would be helped by the hotfix:

    Fatal error encountered in 2XDasm. Exception information is Cannot access a disposed object. Object name: 'CEventingReadStream'.

    There was a failure executing the receive pipeline: "BTAHL72XPipelines.BTAHL72XReceivePipeline, BTAHL72XPipelines, Version=1.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Source: "BTAHL7 2.X Disassembler" Receive Port: "ReceivePortName" URI: "IPAddress:portNumber" Reason: Cannot access a disposed object. Object name: 'CEventingReadStream'.

    The Messaging Engine received an error from transport adapter "MLLP" when notifying the adapter with the BatchComplete event. Reason "Object reference not set to an instance of an object."

    We've been through a lot of troubleshooting with Microsoft Product Support and they did a great job finding an issue and releasing a fix.

  • MSBuild Tasks for BizTalk Continuous Integration

    If you store BizTalk application settings in Enterprise SSO database and use continuous integration in TFS you'll find these two MSBuild tasks useful.

    DeploySSOConfigStore task reads settings from XML configuration file and saves them to the SSO database. The XML is created (exported) using SSO Config Tool which described in previous post. If you need to change configuration settings, just update XML file in the source control and build process will pick it up and propagate changes to your target environment. Using this task is very simple, it accepts just one parameter - location of the XML settings file (and that can be different depending on the build target):

        <UsingTask AssemblyFile="CustomMSBuildTasks.BizTalk.dll" TaskName="CustomMSBuildTasks.BizTalk.DeploySSOConfigStore"/>
        <Target Name="DeploySSOConfigStore" DependsOnTargets="BizTalkDeploy">
            <CustomMSBuildTasks.BizTalk.DeploySSOConfigStore XmlConfigurationUrl="$(SolutionRoot)\$(BuildBranch)\Source\BizTalkSettings\BizTalk.Configuration.xml"/>
        </Target>

    Another MSBuild task is for publishing WCF services - PublishBizTalkWcfServices. It sets up virtual directories (if needed), publishes WCF contracts, creates receive locations. It has comprehensive logging that helps tracking down deployment issues quickly.

        public class PublishBizTalkWcfServices : Task
        {
            private string serviceDescriptionUrl;
     
            /// <summary>
            /// The URL to the services description file.
            /// </summary>
            [Required]
            public string ServiceDescriptionUrl
            {
                get { return serviceDescriptionUrl; }
                set { serviceDescriptionUrl = value; }
            }
     
            /// <summary>
            /// Publishes BizTalk schemas as WCF services.
            /// </summary>
            /// <returns></returns>
            public override bool Execute()
            {
                WcfServiceDescription description = WcfServiceDescription.LoadXml(ServiceDescriptionUrl);
                Publisher publisher = new Publisher();
                publisher.BackgroundWorker = new System.ComponentModel.BackgroundWorker();
                publisher.BackgroundWorker.WorkerReportsProgress = true;
                publisher.BackgroundWorker.ProgressChanged += new System.ComponentModel.ProgressChangedEventHandler(BackgroundWorker_ProgressChanged);
                try
                {
                    PublishingResults results = publisher.Publish(description);
     
                    Log.LogMessage(results.Message);
                }
                catch (Exception ex)
                {
                    Log.LogError(ex.ToString(), null);
                    return false;
                }
                return true;
            }
     
            public void BackgroundWorker_ProgressChanged(object sender, System.ComponentModel.ProgressChangedEventArgs e)
            {
                Log.LogMessage(e.UserState.ToString(), null);
            }
        }

     

    Source code can be downloaded from here http://www.box.net/shared/b7ry6iuw4o

     

Powered by Community Server (Personal Edition), by Telligent Systems