Mohammed Atef’s Technical blog

BizTalk Host Throttling

Introduction

Today i am going to talk about Host Throttling, here i am trying to define Host Throttling and it is types . Really all this information are found at Microsoft BizTalk MSDN, but i am trying to reorganize this information to be quick and useful information.

Host Throttling

Host
Most of the processing that takes place on a BizTalk server occurs within host instance, which is a process running as a Windows service.
Throttling
Manage the use of resources by a host instance process, BizTalk Server utilizes an adjustable throttling mechanism that governs the flow and processing of messages through a host instance
Throttling Benefits
  • Moderates the workload of the host instance to ensure that the workload does not exceed the capacity of the host instance or any downstream host instances.
  • Detects when available resources are being underutilized. If available resources are underutilized then the throttling mechanism allows additional messages to be processed by a host instance
Throttling Configuration
Throttling configuration parameters are set on a per host basis in the BizTalk Server Administration console,
Throttling Configuration Scope
Throttling configuration parameters that are set for the host apply to any or all receive handlers, send handlers, or orchestrations that are housed in the corresponding host instances.

Now lets view two types of Host Throttling

Inbound host throttling Outbound host throttling
Known as
message publishing throttling in BizTalk Server message processing throttling in BizTalk Server
applied to
receive adapters or orchestrations that publish messages to the MessageBox database orchestrations or send adapters that receive and deliver or process messages that have been published to the MessageBox

 

When condition triggered

  • The amount of memory, the number of threads, or the number of database connections used by the host instance.
  • Downstream hosts are unable to process the messages that are published
  • The Message publishing incoming rate for the host instance exceeds the Message publishing outgoing rate
  • The amount of memory, the number of threads, or the number of database connections used by the host instance.
  • The Message delivery incoming rate for the host instance exceeds the Message delivery outgoing rate
  • The number of messages being processed concurrently by the host instance exceeds the In-process messages per CPU
Severity change and actions
  • delay may be implemented when an End Point Manager (EPM) thread receives a batch of messages from the transport adapter
  • EPM submits a batch of messages to be published into the MessageBox database
  • The number of threads that are available to (EPM) is restricted
  • A progressive delay in the processing logic of the host instance is implemented before delivering the messages to the outbound transport adapter or the orchestration engine for processing the messages.
  • The number of messages that can be held by the in-memory queue is limited.
  • The size of the Message Agent thread pool is limited. By limiting the Message Agent thread pool size, the host throttling mechanism effectively reduces the amount of messages that are delivered to XLANG and adapters.
Default configuration
EPM is configured to use 20 threads per CPU the in-memory queue is set to hold 100 messages per CPU
Diagram
image image

I Hope that helped

Advertisements

November 3, 2009 Posted by | Biztalk | , | Leave a comment

Monitor BizTalk 2009 Applications

Introduction

I have found question at Microsoft BizTalk Forums about how to retrieve all BizTalk Application with none started status.

Solution

An Expert MVP called Thiago AlmeidaMVP has answered and advice for using BizTalk ExplorerOM assembly. i have used it to create simple windows application that show all BizTalk application with status in DataGridView.

Understand code

Let us see main code used to retrieve BizTalk application and it is status in the below snippet:

   1: BtsCatalogExplorer catalog = new BtsCatalogExplorer();
   2: DataTable apptbl = CreateAppTBL();
   3: //connection string to BizTalk management database
   4: catalog.ConnectionString = "SERVER=.;DATABASE=BizTalkMgmtDb;Integrated Security=SSPI";
   5: DataRow dr;
   6: foreach (Microsoft.BizTalk.ExplorerOM.Application app in catalog.Applications)
   7: {
   8:     dr = apptbl.NewRow();
   9:     dr["AppName"] = app.Name;
  10:     dr["AppDesc"] = app.Description;
  11:     dr["AppStatus"] = app.Status;          
  12:     apptbl.Rows.Add(dr);
  13: }
  14: return apptbl;

As shown above i am going to retrieve all applications throw BtsCatalogExplorer then use Foreach loop to fetch each application individual and finally fill DataTable with information needed.

now i am going to give you simple description about ConnectionString property found at BtsCatalogExplorer .

ConnectionString : is used for gets or sets the string used to open BizTalk Management database as shown above i have installed BizTalk at local server so i have wrote server=. to point to local server database and BizTalk Database is BizTalkMgmtDbso i have set DataBase with BizTalkMgmtDb value.

you can download full sample from here.

I hope that help.

October 21, 2009 Posted by | Biztalk | , | 1 Comment

Canonical schemas

Introduction

Canonical Schema is doing the same using for Canonical data model which designed to communicate between different data formats, actually this pattern used to decease the cost of integration developments while you are doing enterprise application integration and also it is used heavily in SOA solutions.

Problem

suppose you have five different schemas and you are going to integrate between them what you can do?

Solution without Canonical Schema

you will do integration and mapping between each schema you need to integrate with the other one.so you will do the below design pattern.
canonical schema1

the problem of this solution that you are was a lot of time for mapping and integrating each schema.

Solution Using Canonical Schema

you can create common schema called(Canonical Schema) and at any given time you can translate to or from this schema. it is seems you have only one schema and any other schema can transform to it, i think it will be better for you now.so kindly find the pattern design using Canonical Schema in the image below:

canonical schema2

I hope that helped.

August 19, 2009 Posted by | Architucture, Biztalk, Design Patterns, SOA | , , , | 3 Comments

Covert Complex SQL In clause to a business rule

Suppose you have Complex SQL Stored procedure and you would like to do this Complex SQL operation into BizTalk Business Rule Engine, what can you do??

Let see Example for This…
we need to transform the below SQL script to BRE
Update a Set a.policy=150
from a
Inner join b
on a.id=b.id
Where a.xx=100 and a.yy in (3,10) and b.zz=1
To solve this issue follow this steps
1- create view contain SQL join.
2- Use IF statement in Business Rule
by adding in the if condition a.xx=100 and a.yy in (3,10) and b.zz=1
and put the set action into if condition action
I Hope this Help.

July 29, 2009 Posted by | Biztalk | , , | Leave a comment

Exposing WCF services from orchestrations using BizTalk 2009

Introduction
Today i am trying to make simple BizTalk application for exposing WCF service from Orchestration.
i will explain this example from A to Z and you can download full BizTalk solution by click here .

Problem definition
Our application is used for receive xml file for customer and send this xml file to output folder by updating the customer status to received.

Solution
Solution composed from Three parts as follows :
Part one: Develop Schema and Orchestration and testing it
This part is direct and simple for all BizTalk developer. first i will create new BizTalk project and select add new file after that select schema file and build schema like the following image
schema

then build the solution by specifying strong key and application deployment project attributes.
now we can build the orchestration project.
like the first project select new BizTalk Project and add new item but this time select new Orchestration then build the new orchestration like the below image
Orchestration
all this shapes is very sample and i am sure you can do it but to update the status for customer double click in the MessageAssignment shape and add this two lines of code

//assume the input message called InMsg and output message is OutMsg
OutMSg=InMsg;
OutMSg.Status="Received";

now build the solution by specifying the strong key and application name like the previous project and deploy the solution.

finally go to the BizTalk server application console and extract applications you will find the new application. just you need to define receive and send port and bind these to the generated Orchestration and test you application.

Part Two : Generating Orchestration WCF

Now we are going to generate WCF for our Orchestration so kindly follow this steps

1- select Tools –>BizTalk WCF Service Publish wizard and press next

2- select Service Endpoint radio button and check the below two check boxes

3- select public BizTalk Orchestration as WCF Service

4- press next for all the incoming screens till you reach the finish screen.

now go to IIS you will find new WCF service found named by the orchestration name.

try to browse it now…ohhhh……..not work!!!!!!

to fix this issue go to the BizTalk Application console and open our application you will find new receive location with transport type WCF-WShttp disabled just enable it and go to the orchestration binding screen and select the new generated receive port in it is receive port.

Important note make sure that the application pool user account  for the WCF Service found  in BizTalk Isolated Host Users group.

Now i am sure if you browse this WCF service it will work.

Part three: Consume the WCF Orchestration

lets finish this project it over now. just add console windows project and right click the project name then select add service reference and put the WCF orchestration URL.

open the program.cs file code and add the following code

WCFSOAClient.SOAWCF_soawcf_Port_1Client client = new WCFClient.WCFSOAClient.SOAWCF_soawcf_Port_1Client();
WCFSOAClient.Customer cust = new WCFClient.WCFSOAClient.Customer();
cust.Address = "salmia";
cust.FName="Mohammed";
cust.LName="Atef";
cust.Status="pending";
Console.WriteLine("begin call wcf orchestration");
client.Operation_1(cust);
Console.WriteLine("done! check the output folder");
Console.ReadLine();

now run this console project you will see begin call WCF Orchestration and done! check the output folder and new fill dropped to the out folder contain the following xml.

<?xml version="1.0" encoding="utf-8"?>
<Customer xmlns="http://schema.Schema1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<FName xmlns="">Mohammed</FName>
<LName xmlns="">Atef</LName>
<Address xmlns="">salmia</Address>
<Status xmlns="">Received</Status>
</Customer>

I Hope This Help.

July 8, 2009 Posted by | Biztalk, SOA | , , , | Leave a comment

WCF and WFF vs. BizTalk

Someone might look at Windows Workflow Foundation (WF) and Windows Communication Foundation (WCF) and ask why we are still using BizTalk Server.
Aren’t these exciting technologies replace BizTalk functionality obsolete? Yes today, WF and WCF are foundational platform technologies on which future Microsoft applications will be built upon.
They are both excellent at servicing particular problem areas around unified communication and workflow.
BizTalk is a framework for integration and business process automation that sits on top of the .NET Framework – and it will sit on top of WCF and WF later this year. BizTalk is used to build composite, loosely coupled applications on top of the services that WCF exposes. It provides things such as state management, compensation, access to various legacy protocols and applications and an administration model for operationalizing the applications. BizTalk is a broker that sits between various systems in order to abstract them from one another; it decouples them and eliminates point to point interfaces.
Similarly, while you can build a portal solution using Microsoft Internet Information Services (IIS) and ASP.NET technologies, is it replacing Microsoft SharePoint Server??!! I can build my own messaging solution using WCF and WF, but trying to design, build, and test such a solution takes me away from my primary goal of helping my organization solve business problems.

I Hope this help.

June 29, 2009 Posted by | Biztalk | , , | Leave a comment

AS/400 BizTalk Adapters

Introduction
Microsoft® BizTalk® Server has providing a suite of new technology adapters. These include a new set of adapters for IBM Host Integration scenarios covering DB2, WebSphere MQ, Host Applications, and Host Files. In this post we shall look at WebSphere MQ and Host Files adapters in detail and explain their capabilities and use with AS/400.

WebSphere MQ Adapters
Now WebSphere MQ server on Windows is no longer required to connect to remote Queue Managers on non-Windows® systems. With the Server adapter a component is installed on the MQ computer that interacts with the MQ APIs.Let see MQ adapter architecture with BizTalk in the below image
clip_image002

The above image shows two environments are depicted, the BizTalk Server on Windows Server and an MQ Server which is running on AS/400. The MQSC adapter which enables sends and receives functionality using the chosen client. The MQ architecture for the Client works by receiving and sending messages to and from queues using a channel. A channel is a uni or bi-directional securable mechanism to perform reliable communication and is defined on a given Queue Manager on the server. Between the Client and the Queue Manager on both ends is the Message Channel Agent. It is the agent’s responsibility to control the send and receipt of messages.
Essentially there are three main components to using the Client adapter: MQ Client configuration, BizTalk Server port configuration, and the solution design itself. Let’s cover each of these in turn.

Client Configuration
MQ Client Configuration starts with a choice between the Base Client and Extended Client . The principle difference between the two is that the Extended Client is fully transactional while the Base Client is not. However, the BizTalk Server adapter is still able to guarantee at least once delivery with the Base Client due to the way it uses the client. Essentially, the adapter first retrieves a batch of messages non-destructively and only once they are safely in the MessageBox database are they removed from the queue. This is achieved using a cursor in MQ to enable locking and browsing of a queue without removing the messages. The same process occurs in reverse when sending messages, where the adapter checks the messages are safely on the queue before responding to the BizTalk Message Agent , which will then remove them from the MessageBox database. Because of this, it is possible in failure conditions to cause duplicate messages to be processed and your solution must take this into account if necessary. The Extended client ensures “once and once only” semantics.

BizTalk Server Configuration

There are a multitude of options here and we will consider the use of the most important ones here. The first thing to mention is that configuration is handled differently on Send and Receive. On the Receive side, the transport configuration is achieved through creating a Receive Location while on the Send side a Send Port is created and configured as shown below.
clip_image004

First, the connection between the BizTalk Server and the MQ Server must be defined. This is covered by four properties: channel name, connection name, queue and queue manager name.
The MQ adapter supports full ordered delivery on both the receive and send sides. When both are configured, First In, First Out (FIFO) processing all the way through BizTalk Server is possible .
Now let’s see how the orchestration is implemented as found in the below image
clip_image006

The above image shows an orchestration using the MQ client adapter and manual correlation to implement a solicit-response pattern. First, the orchestration must set the MQ header MQMD_CorrelID to a unique ID and a correlation set initialized containing this context property. Next, the message is sent to an MQ queue. After a period of time, a second message is received on a different queue. The adapter automatically copies this property and the receive port has a following correlation set – the same as that initialized in the send port. This ensures the second message is correlated to the first and the response message delivered to the correct orchestration. Using this approach, two different queues can be used to implement a more flexible approach to the solicit-response pattern.

for BizTalk Adapter for host files you can find this in this post

I hope this help.

June 24, 2009 Posted by | Biztalk | , , , | 2 Comments

BizTalk Adapter for Host Files

This adapter, as its name suggests, enables access to files in a host environment. The Host File adapter enables full CRUD (Create, Read, Update & Delete) access using a SQL-like syntax, implemented in a new managed provider. When used stand-alone the provider enables SQL to be used to provide a familiar ADO.NET programming model for data access. The adapter’s supports usage on both send and receive Ports.

Defining File Structures

There are two constituents to creating a BizTalk solution using the Host File adapter. The first is specific to the host file adapter, a file definition, and the second is the familiar schema generation discussed above in the context of the DB2 adapter. A schema is required to define the file data sent back and forth using XML whilst the file definition is needed to map the file’s contents to XML. Figure below shows this relationship in detail. Taking a look at the file definition first, Host Integration Server introduces a new Microsoft Visual Studio® project type, Host File, enabling the creation of host file definitions. A definition can be created in several ways; manually, through program import or from an existing Host Column Definition (HCD) file.

image

By selected “Add Generated Items” and the File Adapter we can now create the schema to use for transferring data between BizTalk Server and the Host. There are three options, updategrams, SQL SELECT or OS/400 Command. The SQL SELECT option allows the host file system to be queried using a SQL-like syntax. An example select contact,title from customer.

Configuring Ports

Once both the schema and metadata file assembly have been created a port can be configured. As already mentioned, the adapter supports both Send and Receive ports allowing File Polling Receive-side and one-way or solicit-response Send-side. Figure below shows the Receive Location property page for the File Adapter. The connection string can be provided manually or through the Data Source Wizard as with DB2. There are some additional options for host files, the principle one being the Metadata property, which contains the name of the metadata assembly containing the file definition.
clip_image002

Host File adapter does not support batching and returns only a single message for all data returned from a file in one operation. If single record/message processing is required, an envelope schema can be used to split the message up in the receive pipeline processing. This technique may be used whether a Receive Location or solicit-response Send port is being used to return data. The splitting of messages enables parallelism to be achieved allowing BizTalk Server to process multiple records simultaneously rather than having to de-batch them in an orchestration.
The Root element name/namespace are required to create the response messages from the adapter. The purpose of this is to ensure the incoming message (generated by the adapter) will match a particular schema. The SQL Command property enables a SQL query to be specified . Update Command allows the retrieved records to be updated or deleted as required. The principle use for this is to ensure that the same data is not picked up twice due to the polling nature of the adapter. If update is specified, the values for each field to update on each retrieved record must be specified. The URI property value specified must be unique.
Send side, Figure below shows the options. As you can see the configuration is much simpler, with only the connection string and document namespace/root name required with the unique URI. This is because all the information required for accessing the host file is provided in the message schema. It is to this configuration that we turn next.

clip_image004

Although the BizTalk Adapter for Host Files provides a great deal of functionality to process file data through BizTalk Server, there are a couple of limitations that you should be aware of. Firstly, the adapter is not transactional with respect to the Host File system. Although the Host will ensure failure cannot corrupt the file being updated, updates may be lost.
The second limitation is that the adapter does not support dynamic sends. A dynamic send is where the URI containing transport and endpoint address is specified at runtime. The adapter only enables the URI details to be specified on a static Send Port and BizTalk Server will not allow the adapter to be selected on Send Ports defined as dynamic in an orchestration.
I am going to prepare sample for host files adapters with BizTalk in few days.

I hope this help.

June 24, 2009 Posted by | Biztalk | , , | 1 Comment

How BizTalk Maps Work

Introduction

All BizTalk developer have used Map Editor before. But did you ask before how it is working?!! Today I would like to tell you how BizTalk Map Editor is working in this simple post.

How it is working?

BizTalk maps works from Top to Bottom by other way from Target to Source, so BizTalk mapping Target from beginning to end, then Mapping rules are constructed and executed as links are encountered on the output side and data is extracted from the source when a link is encountered in the target at the end data is pulled into the file structure based on the structure of the output schema. I think this is the best way for parsing the Map because sometimes we set some target values without using and source elements.
Also BizTalk Map is designed to generate XML output and the map graphically displayed is actually an XSLT style sheet built to write output into the specific format defined by the output schema, so you can easily generate the *.xsl file containing the map code at any point in the development of your map and look to see how the map rules are being generated.

You can generate xsl Map code by executing the Validate Map function, found by right clicking the map name in the Solution Explorer.

You can try with any BizTalk Map you have and if you have good knowledge about XML and XSLT you can also understand how Maps built?

Understand XSL Map Code

The most component used in the XSL Map are
Variables
when you examine the code XSL Map code you find many unexpected variables. The compiler creates and uses internal variables for its own use, most often to contain the output of functoids. Any time a logical operation produces output that must be consumed by another operation or placed into the target model; the compiler creates an internal variable in which the value can be stored temporarily.
Example

<xsl:variable name="var:v1" select="Address/text()"/>
<shipAddress>
<xsl:value-of select="$var:v1"/>
</shipAddress>

Here in this example we Map the source Address element to the Target ShipAddress.

Compiler-Generated Scripts

we can see C# script for each functoid or for any custom C# script we add using the Script Node.these scripts are found in the CDATA section at the end of the map

Example

<msxsl:script language="C#" implements-prefix="usercsharp">
<![CDATA[
public string DateCurrentDate()
{
DateTime dt=DateTime.Now;
Return dt.ToString("yyyy-MM-dd");
}
]]/></msxsl>

This is very simple C# method used to retun the current date.

Optional Links

If you have element name FName in the source Schema and you directly linked this element to another on called Name int the Target Schema and the two elements are optional so you will find the xsl Map code as follow

<xsl:if test="FName">
<Name><xsl:value-of select="FName/text()"/></Name>
</xsl:if>

but if this two elements(FName,Name) are mandatory you will find the following

<Name><xsl:value-of select=”FName/text()”/></Name>

Using C# code

you can see a lot of calls to C# scripts in this XSL Map.for example you can find the following code for setting the Target Schema AddedDate Element is

<xsl:variable name="var:v1" select="usercsharp:DateCurrentDate()"/>
<AddedDate>
<xsl:value-of select="$var:v1"/>
</AddedDate>

I think it is clear now that BizTalk Map Mainly use simple XSL with C# scripts.

I hope this help.

June 21, 2009 Posted by | Biztalk | , | 3 Comments

BizTalk 2009 Orchestration Dehydration and Persistence Points

Introduction
I am going to describe the Orchestration Dehydration and Persistence Points.this post for understanding why and how BizTalk use these.So let’s go quickly.

Dehydration
We know that BizTalk some times depending on other external application, so it is possible to ask BizTalk for waiting the other external application responses this my consume a lot of resources and memories, Instead of just waiting, the orchestration will dehydrate, and the state of the instance is taken out of memory and stored in SQL Server. BizTalk does this to free up valuable resources
for other processes and special subscription will be created for the dehydrated instance of the orchestration.This subscription will monitor the BizTalk message box for context properties
that match your unique correlation ID. When the subscription comes across that message, the
orchestration will “wake up,” and the state of the instance will be taken out of the database
and reconstituted in memory. The point at which the orchestration will, essentially, come alive
is known as a persistence point.
Persistence Points
The orchestration engine persists the entire state of a running orchestration instance at various points, so that the instance can later be completely restored in memory.
The state includes
1. The internal state of the engine, including its current progress.
2. The state of any .NET components that maintain state information and are being used by the orchestration.
3. Message and variable values.
if a message is received but there is an unexpected shutdown before state can be saved, the engine will not record that it has received the message, and will receive it again upon restarting. The engine will save the state in the following circumstances:

1. The end of a transactional scope is reached.
The engine saves state at the end of a transactional scope so that the point at which the orchestration should resume is defined unambiguously, and so that compensation can be carried out correctly if necessary.
The orchestration will continue to run from the end of the scope if persistence was successful; otherwise, the appropriate exception handler will be invoked. If the scope is transactional and atomic, the engine will save state within that scope.If the scope is transactional and long-running, the engine will generate a new transaction and persist the complete state of the runtime.
2.Debugging breakpoint is reached.
3. message is sent.The only exception to this is when a message is sent from within an atomic transaction scope.
4. The orchestration starts another orchestration asynchronously, as with the Start Orchestration shape.
5. The orchestration instance is suspended.
6. The system shuts down under controlled conditions. in that case, when the engine next runs, it will resume the orchestration instance from the last persistence point that occurred before the shutdown.
7. The engine determines that the instance should be dehydrated.
8.The orchestration instance is finished.

I hope this help…………

June 20, 2009 Posted by | Biztalk | , , | 1 Comment