How to generate a BizTalk request message

What appears to be an issue for many BizTalk developers or newbies, is how to initiate a BizTalk request message.  Most perplexing about this is that BizTalk is not an intuitive interface, which leaves many in the dark as to how or what exactly needs to be done in order to perform a simple select query against a database.  Well, I've wrestled with this issue many times before, and what I've found is that generating your request messages is best served in code.  This is the only means, I find, that permits flexibility even after promoting your application to a production environment.

At the base is the simple question of where does one start?  Well, that's simple enough, just go to your map in Visual Studio, and right-click to perform a "Test map" operation.  If you don't have a map, create a temporary map and make sure you apply the "Generate Instance" setting in the TestMap Input field.  In the Output window, you'll find a link to the sample input file leveraged to generate the test.  Open this sample and voila! You have the xml needed to help generate your request message.

With the sample xml available, you can then create a class that performs the generate request message code.  Modify the xml values such that you can modify them from outside of BizTalk (perhaps in a config file).  Below follows sample code outlining message creation process.

public static XmlDocument CreateMsg(string msgType)
{
     XmlDocument xmlDoc = new XmlDocument();
       StringBuilder strBuilder = new StringBuilder();
            string columnString = string.Empty; \\ ex "column1 as [Sales], column2 as [Pct]
            string filterString = string.Empty; \\ ex "WHERE DATEDIFF(Day, LAST_DATE_WORKED, getdate()) < 8"


            strBuilder.Append("");

            switch (msgType.ToUpper().Trim())
            {
                case "A Specific Message":
                    columnString = ConfigurationManager.AppSettings["myRunTimeColumns"];
                    filterString = ConfigurationManager.AppSettings["myRunTimeWhereClause"];

                    strBuilder.Append(string.Format("{0}",
    columnString));

                    strBuilder.Append(string.Format("{0}",
    filterString));

                    strBuilder.Append("");
                    break;
                default:
                    strBuilder.Append("FirstName, LastName, OfficeNumber, case Code when '850' then Dept else Region end as LocationCode, Email");
                    strBuilder.Append("");
                    break;
            }

            strBuilder.Append("");


            xmlDoc.LoadXml(strBuilder.ToString());

            return xmlDoc;
}

And of course, you would have to declare a message object in your orchestration and initialize it from within the orchestration as such:

MyRequestMsg = HelperNameSpace.MessageConstructorClass.CreateMsg("A Specific Message");

For this example, I populated the BizTalk service config file with new keys under the appSettings node (myRunTimeColumns and myRunTimeWhereClause) to hold the column and where clause values respectively.

OOB BizTalk FTP Adapter Alternative

We tend to get naive when working with BizTalk and its many adapters.  I, for one, admit to operating under such soundless circumstances.  My latest issue has been with the out of the box FTP Adapter Microsoft so graciously included.  My objective was to generate an application that would connect to two separate FTP servers and apply the same file to both.  It just so happens that one of the vendors in this instance, was uncooperative or unable to supply a test account. So, I decided to test my process with the more cooperative of the two vendors and configured my send ports to push both files towards that vendor.

All worked great, and I ran and reran my process dozens of times over.  I passed development, alpha, beta and UAT testing until finally pushing my application to production.  That's when the wheels fell off the wagon.  After I reconfigured my send port to send my files to both vendors, within an hour of post-deployment testing, I soon discovered that the FTP adapter was failing for one of my send ports.  Upon review of the event logs, I noticed that my uncooperative vendor was refusing connection after the initial FTP call.

Digging a little deeper, I turned on the FTP logs made available by the FTP Adapter and found that the Adapter never sends the QUIT command to sever the connection until the host instance was shut down or restarted.  This was insane, I thought.  I tried a number of solutions such as applying an "After Put" command in the FTP Adapter. Sadly these solutions did not work effectively without throwing exceptions and cluttering up BizTalk with failed orchestrations.

So, I eventually gave in and decided to leverage a dynamic port and incorporate C# into the project.  The code declared a new class that would perform folder monitoring based on a value specified from the BizTalk configuration file.  My monitor would listen to the folder and wait for a new file or file modification to occur like such:

public void initiateFolderMonitor()
{
string fullFilePath = ConfigurationManager.AppSettings["FolderPath"];

folderMonitor.Path = fullFilePath;
UploadFilePath = fullFilePath;
UploadFile = ConfigurationManager.AppSettings["FileName"];
FTPAddress = ConfigurationManager.AppSettings["FTPAddress"];
FTPUser = ConfigurationManager.AppSettings["FTPUser"];
FTPPassword = ConfigurationManager.AppSettings["FTPPassword"];
FailureCount = 0;

folderMonitor.NotifyFilter = System.IO.NotifyFilters.FileName;

folderMonitor.NotifyFilter = folderMonitor.NotifyFilter | System.IO.NotifyFilters.Attributes;

folderMonitor.Created += new FileSystemEventHandler(monitoringEventRaised);
folderMonitor.Changed += new FileSystemEventHandler(monitoringEventRaised);


 try
 {
   folderMonitor.EnableRaisingEvents = true;
 }
 catch (ArgumentException)
 {
   stopActivityMonitoring();
 }
}

public void stopActivityMonitoring()
{
 if (FailureCount == 0)
   retryFileUpload();

 if (folderMonitor.EnableRaisingEvents)
   folderMonitor.EnableRaisingEvents = false;

 folderMonitor.Dispose();
 folderMonitor = null;
}

private void monitoringEventRaised(object sender, System.IO.FileSystemEventArgs e)
{
 switch (e.ChangeType)
 {
   case WatcherChangeTypes.Changed:
   case WatcherChangeTypes.Created:
      if (e.Name == UploadFile)
      {
         FailureCount += 1;

         if (FtpUloadFile())
         {
            stopActivityMonitoring();
            removeFile();
         }
         else
         {
            retryFileUpload();
         }
      }
      break;
   default:
      break;
 }
}

private void retryFileUpload()
{
  Thread.Sleep(10000);

  if (FailureCount > 3)
  {
     stopActivityMonitoring();
     return;
  } 
  else if (FtpUloadFile())
  {
     stopActivityMonitoring();
     removeFile();
     return;
  }

  FailureCount += 1;

  retryFileUpload();
}

And naturally, I have code to perform the actual FTP upload like such:

private bool FtpUloadFile()
{
   bool ftpReturn = false;

   try
   {
     System.Net.FtpWebRequest ftpRequest = (System.Net.FtpWebRequest)System.Net.WebRequest.Create(string.Format("ftp://{0}/{1}",
       FTPAddress, 
       UploadFile));
     ftpRequest.Method = System.Net.WebRequestMethods.Ftp.UploadFile;
     ftpRequest.Credentials = new System.Net.NetworkCredential(FTPUser, FTPPassword);

     System.IO.StreamReader streamFR = new System.IO.StreamReader(string.Format(@"{0}\{1}", 
       UploadFilePath, 
       UploadFile));
     byte[] filecontents = Encoding.UTF8.GetBytes(streamFR.ReadToEnd());
     streamFR.Close();
     ftpRequest.ContentLength = filecontents.Length;

     System.IO.Stream requestStr = ftpRequest.GetRequestStream();
     requestStr.Write(filecontents, 0, filecontents.Length);
     requestStr.Close();

     System.Net.FtpWebResponse resp = (System.Net.FtpWebResponse)ftpRequest.GetResponse();

     switch (resp.StatusCode)
     {
       case System.Net.FtpStatusCode.ClosingData:
          ftpReturn = true;
          break;
       default:
          ftpReturn = false;
          break;
     }

     resp.Close();
   }
   catch (Exception) { }

   return ftpReturn;
}

So, the above code gets initiated from my orchestration before the send shape. You have to declare a variable to hold the new class. Then, within an expression shape, you set the Folder monitoring in motion like so:

//Initiate Orchestration variable
monitorObj = new my.Helpers.MonitoringClass();
monitorObj.initiateFolderMonitor();

//Initiate the dynamic Send port
DynamicSendFilePort(Microsoft.XLANGs.BaseTypes.Address) = @"file://" + ConfigurationManager.AppSettings["FolderPath"] + @"\" + ConfigurationManager.AppSettings["FileName"];

Finally, in the orchestration, you can either call the disable monitoring method, or include code in your class to timeout the monitoring piece to release those system resources.

How to clone a SharePoint 2010 Hyper-V VM

If anyone has ever used Microsoft's Hyper-V Server, you've discovered that the interface is a little difficult.  For one, you're presented with no GUI, which can be intimidating to some.  In any case, I was able to spend a little time to uncover a few steps that could help towards cloning a standalone SharePoint instance hosted on VM.  Let's begin...

Your first step involves logging into the Hyper-V base.  On the base, what we want to accomplish is to create a copy of the VHD file containing the SharePoint 2010 instance.  For some, you may log into the base and find that there are no visible windows.  This little obstacle can be overcome by calling the Task Manager window; simultaneously press the Control-Shift-Esc keys.  That should help ease the nerves a bit.

With the Task Manager open, click the File->New Task(Run) option from the file menu.  This will popup a "Create New Task" window similar to what you'd find when you recall the run command in a normal windows environment.  At this point, you want to enter the following command “c:\Windows\System32\sconfig.cmd” to bring up the Server Configuration User window.  You'll want to open the "Create New Task" window again to open an instance of the command prompt as well.

Now, you need to know where the VHD file is located that you wish to copy.  If you don't, a quick way of determining the location is by connecting to the base image from another machine with Windows Explorer and looking under the VM configuration file repository normally located under "C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines."  Open the available XML files until you find the one containing a reference to the file you want to copy.  With this, you will have the path to the VHD file.

On the base machine, go to the DOS prompt and navigate to the location of the VHD file.  Perform a DOS command to copy the file to wherever you wish.  Once copied, you can paste the file to the Hyper-V base image of your choice.

With a copy of the VHD file on your target base image, you'll need to log into a server that hosts Hyper-V Manager.  From Hyper-V Manager, navigate to the target base machine and begin the process of creating a new "Virtual Machine."  Assign a name to the new "Virtual Machine," and allocate an adequate amount of memory.  Configure the networking credentials for the VM before finally selecting the location of the VHD file copied over.

With those steps, you will have created a new VM instance that is an exact clone of the prior VM instance.  But we're not done just yet.  Now we want to rename the new VM instance, or suffer through duplicate name conflicts.  However, because we're also working with SharePoint, we need to modify the Alternate Access Mappings in Central Administration before anything else occurs.

Connect to the new VM instance from Hyper-V Manager.  Log in under a local administrator account and open "Central Administration."  When and if you're prompted for credentials, use the local administrator credentials.  Navigate to "Configure Alternate Access Mappings" and configure the URLs, leveraging the prior VM instance name, to point to localhost.  

One other SharePoint centric step required before officially renaming the server is to run an stsadm command.  There may be a powershell command you can run, but I haven't performed enough research to determine what that would be.  So you need to open a command prompt as administrator by explicitly selecting the "run as administrator" option.  In your command prompt, change the directory to "c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\bin" or wherever the bin folder for 14-hive is located.  Run the command "Stsadm -o renameserver -newservername {YourNewServerName} -oldservername {YourOldServerName}"  to tell SharePoint that a server name change is in effect.

Now, let's bring up Server Manager to change that pesky server name by selecting Start->Administrative Tools->Server Manager.  From the Server Summary section, select "Change System Properties."  Click the "Change" button from the "Computer Name" tab and change the server name in the available text field.  You'll be asked to restart the VM.  Allow the server to restart.

Not quite done, but we're almost there.  Log back into the VM instance and open "Central Administration" again.  Go to the "Manage the farm administrators group" under the Security section.  You'll more than likely see accounts that reflect the server name of the prior VM instance.  You'll need to edit those accounts so that their names display the appropriate server name.

Voila!  These steps should get you through cloning a VM instance with SharePoint 2010.  A few assumptions are made with this tutorial and that is that it is assumed that SQL Server resides on the same VM instance.  Also assumed is that you have knowledge of the basic DOS commands required of this tutorial.  And finally, it is assumed that you have full access to the Hyper-V environment.

Generating Flat Files With BizTalk 2009

Again, I was asked to produce a simple comma delimited file with BizTalk.  Easy, right? Well, it is once you know what you're doing, but for those of us (myself included) who rarely are asked to produce these flat files, it can be a bit challenging.  For one, we're used to XML formatting and generally writing to a database or simply producing an XML file.

So I can't stress enough how convenient it is leverage the Flat File Schema wizard.  Only catch is, you need a sample file to get the wizard rolling along. Just create a quick sample file with one row of data and the columns you need, and don't forget to add a carriage return after the last column.

What you get next is another window asking for a little clarification.  Little tricky here.  Ultimately, what you would like is a csv file that applies a carriage return after each row of data.  So select the entire row including the carriage return. Following that, make sure you are defining the record by a delimiter.  A screen will appear asking such, so populate the "child delimiter" field with a carriage return/line feed delimiter.

Moving right along. The next screen is very important. Here, you're defining the name of that root element (not that important) and the element type (important).  Select the Record element type, click next.

You'll go through the same screens for the next set of elements. Only difference here is that you won't be selecting the carriage return when asked to "Select Document Data," and your delimiter will be something else (usually a comma for comma delimited files). And of course your "Child Elements" will be defined as "Field element."  One other thing to note, ensure that your root element has the "child order" property set to "postfix."

From here your only other requirement is to define a send pipeline.  Why you ask? Because the send pipelines format your data according to the specifications defined within your schema.  But not to worry, all you have to know is that in your pipeline file, you simply need to drag a "Flat file assembler" on to the Assemble shape.  Once there, define the "Document schema" to that of what was defined by the Flat File Schema wizard.

Voila! You're ready to generate csv files....Unless of course you wanted to include column headers.  I can't stress enough how difficult it was to find resources that explained how these header schemas work.  Oh sure, define another schema with your column names and your done, right?  Not for all of us.  What I didn't understand was that even after I defined a header schema, 2 things were occurring; empty string values for column names and no carriage return.  

So I set off to find what was wrong with my definition.  What I found was that, I needed to define either the "Fixed" or "Default Value"  property for each element field.   But before I get into that, let me note that defining a header schema is as simple as taking a copy of your document schema and manipulating the values for Fixed or Default and ensuring that a PostFix value is defined for the child order of the root record element.  With that, your header values will print and your carriage return will push the first row of data below the column headers.  Now you're done. Well, almost.  Go back to your Send Pipeline and define the Header Schema property with the newly created Header Schema and now you can deploy and enjoy your BizTalk application.

BizTalk Orchestrations and their respective ports

It seems that every time I have a new BizTalk project land on my desk, my brain turns to mush.  My most recent project is a case in point.

So I was asked to create a simple project that would take data from two disparate systems and merge/clean the data, before finally outputting said data to a simple XML file.  "No problem!" or so I thought.  Recalling my BizTalk acronym SOME (Schemas - Orchestrations - Maps - ESB Itineraries), I was able to quickly produce a project that would perform the task required.  Everything compiled ok.  I was able to deploy successfully (or so I thought).

I only incurred one minor issue, I'd forgotten how to configure my send ports, leveraging "Specify later" binding, to produce an xml file to a directory.  "This can't be" I thought, "I've created several other send ports similar to what I need now."  Thus, panic set in once again.  So I did what any other would do in this event...I reviewed my past to uncover the secret behind properly configuring my send port.  Only problem was that the answer was not as blatantly obvious as I would have preferred. 

"Oh my god! How do you configure a simple send port?"  Well folks, the answer is rather trivial, but annoying nonetheless.  Within your orchestration, you will recall having to define a Port Type.  Some important properties to configure within this type is the "Type Modifier."  Type Modifier should be set to public so that it becomes viewable via Administration Console.  Next you should take note of the Port identifier property for the Port, not to be confused with the Port Type.  This property will prove useful when configuring your send port via Administration Console. 

So you've set your "Type Modifier" to public and you've successfully deployed your solution.  Great!  Now all you need is to configure a send port on the Administration Console.  So select the Application, and create a new send port of Transport Type "File."  Beautiful!  Now BizTalk should produce files for me to the directory specified....right?  Not quite, and here's where I panicked a bit.  At this point, you've declared a send port at the BizTalk level, and you've declared a send port at the project level.  So how do you tell BizTalk to pair the two?  Well, it just so happens that when you right-click the Application in Administration Console, and select "Configure," you are presented with a window displaying the artifacts available within the app.  You'll find both the orchestrations, and send/receive ports created.  Your final step here is to select the orchestration that produces the send port and pair it with the send port you configured via Administration Console.

Voila! 

First Annoyance Solved For Sharepoint 2010

So I was asked to begin the process of converting our Sharepoint 2007 apps to 2010 solutions.  My first thought was, "Oh Great! I forgot to study."   Once I got over the initial shock, I began tackling the assignment with clueless abandon.  I must have attempted to convert 7 to 10 different solutions with the out of the box Sharepoint conversion wizard.  Attempted, but never realized.

Needless to say, it didn't go well.  With little to no help via the web, I scrapped the idea and was momentarily defeated.  But it wouldn't be my style to simply give up.  So off I went, on an adventure to the center of the Sharepoint universe (or at least the edge of my seat).

Long story short, I created an empty sharepoint project with Visual Studio 2010, selected Sandboxed solution and added all my solution files to the project.  All well and good, my project compiled and I was able to deploy to Sharepoint...except.  I couldn't add my webpart.  I'll spare you with the troubleshooting effort and ambiguous exception messages.  To get around my issue, I discovered a few articles stating that certain solutions need to be Farmed, not Sandboxed.

Easy enough I thought, so I began looking through every VS menu item and right-clicking every conceivable object in site to convert my solution to Farm.  No luck, there was a suggestion to open the project file and hand enter the xml node, but I ran into trouble here, plus I was confident that MS had an easy method for converting these silly Sharepoint projects, I just had to dig.  Well, wouldn't you know it, I found the solution.  Just view the project properties by selecting it in Solution Explorer and setting the "Sandboxed Solution" property to False.  So there you have it.  Simple solutions that could leave you feeling silly once you find them or worse yet, feeling like an idiot when your non-technical wife points it out to you.

BizTalk for beginners

It's been some time since my last post, for no other reason than I've gone from SharePoint development to BizTalk related activities in recent months.  In any case, let's get to it.

For this first BizTalk related post, I just want to create a beginner's level guide to developing BizTalk projects.  So what are the terms floating around BizTalk; Orchestrations, Itineraries, Maps, Schemas, Send/Receive Ports, Messages, etc.  Now, I can go over the terms for everyone, but it's not necessary to review what's been published hundreds of times already. My focus is to help developers adjust their current mindset to become open to the idea of project compartmentalization.  I have found that it is easiest to consider every aspect of a BizTalk project as an object.  And as a rule of thumb, work from the ground up, meaning that the development effort should begin with the most basic/independent "objects" first.

From a developers standpoint, it's important to become familiar with Schemas first and foremost.  Why? For no other reason than it is how your message content will be defined for specific operations and they are the absolutely most basic aspect of the BizTalk project.  Schemas are defined in Xml format.  Generally, the top elements define the message operations, such as select or delete operations, and their children define the data processed by these operations.  So it is possible to have a field defined several times within one schema.

A big problem I first ran into during development was that I had no idea how to structure my incoming message for unit testing.  Luckily, (and I found this out after many trials) Visual Studio has the ability to generate an instance of a message based on a supporting schema (Just right-click the schema and select Generate Instance).  This comes in handy when running tests againsts maps....

So how do we transfer information from one Message/Schema to another? Maps (There are other methods, but this is just a beginner's guide).  When you create a new mapping file, you are asked for a source and destination schema.  This is where the Schemas you generated come in handy.  Define a source and destination and leverage the mapper as much as possible to map fields from one schema to the other.  There may be fields that require some massaging before transmitting to their respective destination schema fields, and this is where it becomes important to familiarize yourself with functoids.

Functoids are the tools leveraged within a map to translate data.  They come in many forms; String functoids, Mathematical functoids, Date/Time functiods, etc.  Drop these on your map and define their input/output fields, as well as any other information required of them (such as actual C# code for instance).

In review, create your schemas first, then create your Maps.  Leverage Visual Studio to generate an instance of your source message/schema so you can test your maps more precisely.  Don't forget the functoids to translate your data for the destination system.

Ok. Ports.  You have send and receive ports.  As you may have guessed, ports define what data is transmitted into or out of the BizTalk environment.  But it's not enough to discuss ports alone, because ports have several working parts; Port Types, Operations, Communication Patterns, Message Types, Message Directions, Binding.   Where do we start? Start with Port Types.

To discuss Port Types we should first discuss Communication Patterns.  Communication Patterns define the direction in which the data is to be transmitted by the port.  There are two options; One way or Request-Response.  A one way communication pattern is best defined as a pattern that does not require a formal handshake, whereas a Request-Response pattern does.  To best illustrate this, think of the task of running a select query against a database.  The handshake in this instance occurs when the database receives the select query and returns a resultset back.  Conversely, the one way pattern is illustrated as a push event.

Operations are simply actions that define what activity can occur over a port.  So let's say you have a port that connects a SQL database to BizTalk, the operations can be select, insert, and update operations.

Message Directions define the how the message content is used. It can be either as a request or a response, much like you have requests and responses over http.

Message Types are simply the operations defined within the schemas.  As mentioned earlier, schemas are defined in Xml format.  Their top level elements are generally used to define specific operations and their children define the data processed by these operations.

So Ports need a Port Type assigned to them.  Once a port type is selected for a port, the communication direction is automatically selected for it.  For one way ports, this is fine, but on Request-Response ports, you must be cautious that the comm direction is the one you intended, otherwise you may have to manually select the comm direction.  To help you decide, you must determine whether the port will send data and expect to receive data in return, or vice-versa. Bindings on the other hand can be defined either through Visual Studio or the BizTalk Administration console.  In either case, the bindings are basically the system defined connection protocol.

With all of these defined, the Ports then only allow messages that match the message type defined within their Port type. Not much left to be said about that, so on to the next item...

Messages are simple as well. Consider them the data packages defined by an operation instance within a schema.  Thus, a Message Type must be selected for each message.  Again, not much left to say about messages.

Next up are orchestrations.  With Schemas, Maps, Messages and Ports defined, the orchestration binds these objects all together.  Now within the Orchestrations, you setup your Messages, Ports, Port Types, etc. Basically, everything can be setup from within an orchestration file, but you're better off creating your Maps and Schemas first and adding references to them from within the orchestration file.  All other objects are fair game to create within the Orchestration.

Now, tips.  I've found that it is best to have two separate orchestrations.  One to act as a container for your Port Types, and the other to handle the rest.  Why, because you may have several orchestrations within a project file, so it becomes easier to maintain your port types from one source file and reference those types from the other orchestration files.  Also, I've found that my orchestrations generally leverage similar port types when contained within a project.

Well,  that's it for now.  Hopefully, I'll pick up a few more tips before Microsoft buries BizTalk, in which case, I may have to turn to AppFabric.