GBM
GBM

Reputation: 155

SSIS C# Web API UploadString Identify Bad Data

I have an SSIS package which sends data as a string to an external company's Web API via a Post request, and the string is a pipe delimited one derived from thousands of rows retrieved by a SQL query - the data from the SQL table is them passed as a string to a User variable called User::RequestBody, which is then referenced in the Script Task for the webclient.UploadString call. The problem I am facing is that, if there's a single bit of bad data amongst the thousands rows, the API request is rejected. I then have to try and identify which of the data is bad and as you can imagine, it is pretty time-consuming. Below is the code I have in the Script Task of the SSIS package;

public void Main()
        {

            string username = Convert.ToString(Dts.Variables["$Project::DestinationUserName"].Value);
            string password = Convert.ToString(Dts.Variables["$Project::DestinationPassword"].GetSensitiveValue());
            Uri DestinationURI = new Uri((string)Dts.Variables["$Project::DestinationSite"].Value);
            string stringToSend = (string)Dts.Variables["User::RequestBody"].Value;
            if (stringToSend != "")
            {
                string result;
                using (WebClient webClient = new WebClient())
                {

                    var bytes = Encoding.UTF8.GetBytes(username + ":" + password);
                    var auth = "Basic " + Convert.ToBase64String(bytes);

                    //NetworkCredential myCreds = new NetworkCredential(username, password);
                    //webClient.Credentials = myCreds;
                    webClient.Headers[HttpRequestHeader.Authorization] = auth; 
                    webClient.Headers[HttpRequestHeader.ContentType] = "text/plain"; 
                    ServicePointManager.SecurityProtocol = ServicePointManager.SecurityProtocol | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; beyond the standard SSL3 and TLS
                    try
                    {
                      result = webClient.UploadString(DestinationURI, stringToSend); 

                    }
                    catch (WebException ex)
                    {
                        Dts.Events.FireError(0, "", "UnableToSendData: " + ex.Message.ToString() + ex.StackTrace, string.Empty, 0);
                        return;
                    }
                }

            }
            Dts.TaskResult = (int)ScriptResults.Success;
        }

Below is a sample of 2 records for reference (\n is the start and ending of each data row), but of course we usually send about 9,000 records daily;

"123456|123456|TestUser1|Adam|Mphil/PhD History (PT) Year|3|Department of History||P||||XK|Information not sought|||123456|[email protected]|||||V1ZR-Y||HISs|H|M|1998-05-21||PGR|||0777XXXXXX|||2021-09-19|2022-09-10| |Test NOK: 58 Wiltshire Ave, HA8 5DG: 0208XXXXXX|||||\n678910|678910|TestUser2|Smith|Mphil/PhD Near and Middle East (PT) Year|12|Other Middle East||P||||XK|No known disability|||678910|[email protected]|||||T6ZT/D-Y||OMEs|H|M|1990-03-15||PGR|||079XXXXXXXX|||2022-09-26|2023-09-25| |Test NOK2: 45 Britannia Street, MK11 2ZQ: 0207XXXXXX|||||\n"

I must add, when we send the Post Request to the Web API, we log the response to table in SSMS - the table simple logs the request response, and in this case it's a '400 Bad Request' response. Is there any way to identify where exactly the Request falls over in a case where thousands of records are being sent, as in, can the offending records be identified in a simple way? I'm thinking maybe attempting to send the data one-by-one and if the Post Request encounters a record with bad data, it can be flagged.

Upvotes: 0

Views: 55

Answers (0)

Related Questions