BizTalk incorrect request (multipart form data) part 2

Question

After our first attempt at solving this we decided to go with Greg Forsythe's code. The C# code in its entirety is down below. I am also showing the request from the Postman (which works perfectly) and the one from BizTalk (which doesnt work). But as you can see we are very very close at solving this one. I must also mention that we do not have any mime component any more in the pipeline.

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(Microsoft.BizTalk.Component.Interop.IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            return CreateMultipartMIME(pContext, pInMsg);
        }

        private IBaseMessage CreateMultipartMIME(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Boundary
            string separator = Guid.NewGuid().ToString();

            // New message
            IBaseMessage outMsg = pContext.GetMessageFactory().CreateMessage();
            outMsg.Context = PipelineUtil.CloneMessageContext(pInMsg.Context);

            // Create messagepart (BodyPart)
            IBaseMessagePart multiPart = pContext.GetMessageFactory().CreateMessagePart();
            multiPart.ContentType = "text/plain";
            multiPart.Data = CreateMIME(pInMsg, separator);

            // Add messagepart as BodyPart
            outMsg.AddPart("multiPart", multiPart, true);
            outMsg.BodyPart.ContentType = string.Format("multipart/form-data; boundary={0}", separator);
            outMsg.Context.Promote("ContentType", "http://schemas.microsoft.com/BizTalk/2003/http-properties", string.Format("multipart/form-data; boundary={0}", separator));

            return outMsg;
        }

        private Stream CreateMIME(IBaseMessage pInMsg, string separator)
        {
            Stream mimeStream = new VirtualStream();
            TextWriter writer = new StreamWriter(mimeStream);
            // MIME-header

            // form-data: Bodypart
            writer.WriteLine(string.Format("--{0}", separator));
            writer.WriteLine("Content-Disposition: form-data; name=\"data\"; filename=\"Dk_{5AA9547A-E103-40CD-A2F4-6B77F010E570}.xbrl\"");
            writer.WriteLine("Content-Type: application/xml");
            writer.WriteLine();
            // Read BodyPart stream
            using (StreamReader reader = new StreamReader(pInMsg.BodyPart.GetOriginalDataStream()))
            {
                while (!reader.EndOfStream)
                {
                    writer.WriteLine(reader.ReadLine());
                }
            }
            // form-data: email
            writer.WriteLine(string.Format("--{0}", separator));
            writer.WriteLine("Content-Disposition: form-data; name=\"email\"");
            writer.WriteLine("Content-Type: text/plain");
            writer.WriteLine();
            writer.WriteLine("fsfsd@fsdf.com");
            writer.WriteLine(string.Format("--{0}", separator));
            writer.Flush();
            mimeStream.Position = 0;

            return mimeStream;
        }

And now the request from Postman (working):

Content-Type: multipart/form-data; boundary=--------------------------035085236562027409735938
Content-Length: 299420

----------------------------035085236562027409735938
Content-Disposition: form-data; name="data"; filename="Dk_{5AA9547A-E103-40CD-A2F4-6B77F010E570}.xbrl"
Content-Type: application/xml

<?xml version="1.0" encoding="utf-8"?>
  
<!- ---- Lots of removed xbrl ---- ->

</xbrli:xbrl>
----------------------------035085236562027409735938
Content-Disposition: form-data; name="email"
Content-Type: text/plain

test@test.com
----------------------------035085236562027409735938--

And finally our current request from Biztalk to receiving system:

--26f685f3-5804-44dd-b76b-b3de6391bda1
                      

                                                    
Content-Disposition: form-data; name="data"; filename="Dk_{5AA9547A-E103-40CD-A2F4-6B77F010E570}.xbrl"
Content-Type: application/xml

<?xml version="1.0" encoding="utf-8"?>

<!- ---- Lots of removed xbrl ---- ->

</xbrli:xbrl>
--26f685f3-5804-44dd-b76b-b3de6391bda1
Content-Disposition: form-data; name="email"
Content-Type: text/plain

fsfsd@fsdf.com
--26f685f3-5804-44dd-b76b-b3de6391bda1

The below error message i everything I receive back:

A message sent to adapter "WCF-WebHttp" on send port "A_Sxxx_REST" with URI "https://xxx/xxx" is suspended. 
 Error details: System.Net.WebException: The remote server returned an unexpected response: (400) Bad Request.
{"error":{"status":400,"message":"HTTP 400 Bad Request"}} 
 MessageId:  {DA019D0C-3C31-4C2C-BEA4-FBFCDF646984}
 InstanceID: {B44ED7E8-67EE-4C35-BAC8-3D81553D6AA2}

So, does anyone know why the first part of the Postman request does not appear in the solution that we have programmed?

Answer 1

Solution

After literally working with this issue for three full weeks we have finally managed to solve the issue. For anyone needing a quick and gritty solution, our experiences, final code and some explanations are below.

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(Microsoft.BizTalk.Component.Interop.IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            return CreateMultipartMIME(pContext, pInMsg);
        }

        private IBaseMessage CreateMultipartMIME(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Boundary
            string separator = "231422701264181268539385ABC";

            // Create new message
            IBaseMessage outMsg = pContext.GetMessageFactory().CreateMessage();
            outMsg.Context = PipelineUtil.CloneMessageContext(pInMsg.Context);

            // Create messagepart (BodyPart)
            IBaseMessagePart multiPart = pContext.GetMessageFactory().CreateMessagePart();
            multiPart.Data = CreateMIME(pInMsg, separator);

            outMsg.Context.Write("HttpHeaders", "http://schemas.microsoft.com/BizTalk/2006/01/Adapters/WCF-properties", string.Format("content-type: multipart/form-data; boundary={0} \r\naccept: application/json\r\n", separator));

            return outMsg;
        }

        private Stream CreateMIME(IBaseMessage pInMsg, string separator)
        {
            Stream mimeStream = new VirtualStream();
            TextWriter writer = new StreamWriter(mimeStream);
            // MIME-header

            // form-data: Bodypart
            writer.WriteLine(string.Format("--{0}", separator));
            writer.WriteLine("Content-Disposition: form-data; name=\"data\"; filename=\"Dk_{5AA9547A-E103-40CD-A2F4-6B77F010E570}.xbrl\"");
            writer.WriteLine("Content-Type: application/xml");
            writer.WriteLine();
            // Read BodyPart stream
            using (StreamReader reader = new StreamReader(pInMsg.BodyPart.GetOriginalDataStream()))
            {
                while (!reader.EndOfStream)
                {
                    writer.WriteLine(reader.ReadLine());
                }
            }
            // form-data: email
            writer.WriteLine(string.Format("--{0}", separator));
            writer.WriteLine("Content-Disposition: form-data; name=\"email\"");
            writer.WriteLine("Content-Type: text/plain");
            writer.WriteLine();
            writer.WriteLine("fsfsd@fsdf.se");
            writer.WriteLine(string.Format("--{0}--", separator));
            writer.Flush();
            mimeStream.Position = 0;

            return mimeStream;
        }

Now, what makes this code actually work? Several things, actually. Our first issue was that the receiving system sends a JSON confirmation back. But our http headers clearly states that we only accept application/xml (the http Accept header) content. So we tried to set the Accept header and managed to get it set correctly. But the prolem is now that our Content-Type header retorted to the default one, Content-Type: Application/xml. So to get both of our headers correct, we had to combine them into one code line. Now the headers are clearly looking good, the Content-Type is set as form-data/multipart and Accept is stated as application/json,application/xml.

outMsg.Context.Write("HttpHeaders", "http://schemas.microsoft.com/BizTalk/2006/01/Adapters/WCF-properties", string.Format("content-type: multipart/form-data; boundary={0} \r\naccept: application/json\r\n", separator));

Postman:

Biztalk:

The next problem to solve was the boundary separator. I pretty much stumbled upon this problem when comparing views in Fiddler.

Postman:

Biztalk:

The difference is obvious now BUT we actually sent a rather large file so we never did see the last part. We always believed that everything was okay and that the difference was in how Biztalk and Postman sends files. We were quite wrong. It was first when we used a minimalistic file that we finally could see that the email section which is supposed to be in a separate form is actually coming under the data part. No wonder our receiving system always returned 400 - bad request message. We finally did understand that we are not setting our boundaries correctly.

According to this SO MIME boundary article, the MIME separator must look a certain way for the message to actually be considered multipart. We were very much misled by Fiddler since we always compared the raws from Postman and Biztalk. They looked pretty much the same. But there were differnces that we did not account for.

Boundaries must be used like this:

--boundary
1. body-part
--boundary
2. body-part
--boundary
3. body-part
--boundary--

The preceeding -- is mandatory for every boundary used in the message and the trailing -- is mandatory for the closing boundary (close-delimiter). This signifies that the message part is the last one and that the message should end here.

After fixing this we ended up with this:

Biztalk:

Finally it looks good with the form data and multipart actually looking good. The last small issue we fixed was that the receiving system expects a file name. That was easily fixed with the code of line:

writer.WriteLine("Content-Disposition: form-data; name=\"data\"; filename=\"Dk_{5AA9547A-E103-40CD-A2F4-6B77F010E570}.xbrl\"");

Now, some final words regarding the physical send port. We are using a WCF-WebHttp port and have no outbound http headers set in the port transport properties. It is empty and should be that way since we set everything in code.

It has been one heck of a journey these three weeks. I really hope this post will help someone in the future. Thanks to everyone who tried to assist in any way. Much appreciated.