About Zombies and Convoys...

This week was Load Testing week for our latest set of Interfaces. Incidentally, this was our first set of Interfaces which implemented Uniform Sequential Convoys in their Orchestrations. The implementation was more or less like my previous post. Meaning the convoys had a timeout mechanism, which lead the Orchestration to completion, if no new messages would arrive for a period of time.

So far - so good. But under a realistic amount of load, we observed that almost 1 out of every 5-6 messages would fail to go through to the destination. The HAT would show the following:


Completed with discarded messages

Of course the 1/5 is a very subjective number, depending upon the actual convoy implementation in question. After some snooping around (mostly at Lee Graber’s blog), I figured out that the source of the problem. But no amount of the aforementioned snooping yielded any easy and/or elegant solution to this little pickle.

What happens is that the control might pass to the delay branch of the Listen shape where we implement some logic to escape the loop. During this time interval, if a message arrives at the MessageBox, Biztalk sees our Orchestration as “Running” and since our Orchestration is a match for the Message’s subscription, Biztalk assigns this orchestration to take care of this message. Now our orchestration is blissfully unaware that it has a message or messages to take care of, and runs to completion. Hence, these message(s) are left in the message box with no one to process them or in other words in a Zombie state.

In my opinion, the first thing to do in this situation is (and I am not being facetious here) to verify that your correlation set is actually exact enough for your business scenario. For e.g. if we are transmitting patient medical records, we need to correlate only the same patients and not ALL patients. In other words don’t correlate JUST on MessageType if you can correlate on MessageType AND PatientID.

Secondly set your delay interval to an appropriate value. Setting it too low creates more such race timespans, per unit time. This will reduce your Zombie chances, but won’t give you a 100% success guarantee.

Thirdly have a proper plan to deal with these Zombies according to your business scenario. Retransmit? Discard? What?

Finally ponder about the “Receive Draining” pattern mentioned in Lee Graber’s blog, and if you have a nice elegant way of doing it, tell me about it FIRST!


Comments

Anonymous said…
Some time ago I created a test implementation to avoid those annoying zombies with non-deterministic serial convoys and I guess it was a lot closer to being bulletproof than anything I've seen yet. But I've never had the time to finish my tests and I didn't really wanted to publish it 'as is' on my blog.
If you are interested and like to test it, go ahead and drop me an e-mail...

Br,
Gregory (gregory@eai.be)
May 31, 2005 6:13 AM
Post a Comment

Popular posts from this blog

Sending Messages as FORM Variables with the HTTP Adapter

Image
Here's a quickie. If you want to send a message, as a FORM Variable in a HTTP request, this is what you do: 1) Set the Content Type property of the HTTP send port, to application/x-www-form-urlencoded . 2) Create a custom pipeline and plug-in a Custom Pipeline Component, that pre-pends Variable-Name = in front of the message, and url encode it. e.g: ClaimPacket= 3) Use this pipeline with your HTTP send port, and you are done! Post a comment for a sample. Update : Screen shot of the Content Type property:
Read more

Using recursion in the In-line XSLT Template type Scripting Functoid

People using In-Line XSLT Templates in their scripting functoids often run into the need for looping, WITHOUT using an <xsl:for-each> . An example in the BizTalk context would be, if you had a comma separated string in the source schema and wanted to tokenize it into multiple destination schema elements. The following XSLT template does that for you. <xsl:template name="TokenizeCommaSepString"> <xsl:param name="stringToTokenize" /> <xsl:param name="destinationElementName" /> <xsl:if test="$stringToTokenize != ''"> <xsl:choose> <xsl:when test="contains($stringToTokenize, ',')=0"> <xsl:element name="{$destinationElementName}"> <xsl:value-of select="$stringToTokenize" /> </xsl:element> </xsl:when> <xsl:otherwise> <xsl:element name="{$destinationElementName}"> <xsl:value-of select="substring-be
Read more

Uniform Sequential Convoys in BTS 2004 using Correlation

This orchestration is a small primer if you want to implement a "Sequential Convoy " with BizTalk 2004. You would want to do this if: a) You have to force messages coming in at a particular port to correlate together and get processed by the same orchestration instance. (Correlation with multiple ports is a topic of a subsequent post) b) You want your output messages to be delivered in the same order that the orchestration got them. c) Some pipeline components you use, take one message but give out multiple messages, but you want to process these multiple messages together. For eg: the HIPAA and HL7 disassembler components. ...and probably some more scenarios that I can't think of right now. The attached orchestration correlates messages w.r.t ReceivedFileName (think about the HIPAA 835 scenario mentioned above). You can use whatever correlation parameters you desire. TIP: Whatever properties you choose to correlate the incoming messages with, make sure tha
Read more