Reputation: 155
I have an SSIS package which sends data as a string to an external company's Web API via a Post request, and the string is a pipe delimited one derived from thousands of rows retrieved by a SQL query - the data from the SQL table is them passed as a string to a User variable called User::RequestBody, which is then referenced in the Script Task for the webclient.UploadString call. The problem I am facing is that, if there's a single bit of bad data amongst the thousands rows, the API request is rejected. I then have to try and identify which of the data is bad and as you can imagine, it is pretty time-consuming. Below is the code I have in the Script Task of the SSIS package;
public void Main()
{
string username = Convert.ToString(Dts.Variables["$Project::DestinationUserName"].Value);
string password = Convert.ToString(Dts.Variables["$Project::DestinationPassword"].GetSensitiveValue());
Uri DestinationURI = new Uri((string)Dts.Variables["$Project::DestinationSite"].Value);
string stringToSend = (string)Dts.Variables["User::RequestBody"].Value;
if (stringToSend != "")
{
string result;
using (WebClient webClient = new WebClient())
{
var bytes = Encoding.UTF8.GetBytes(username + ":" + password);
var auth = "Basic " + Convert.ToBase64String(bytes);
//NetworkCredential myCreds = new NetworkCredential(username, password);
//webClient.Credentials = myCreds;
webClient.Headers[HttpRequestHeader.Authorization] = auth;
webClient.Headers[HttpRequestHeader.ContentType] = "text/plain";
ServicePointManager.SecurityProtocol = ServicePointManager.SecurityProtocol | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; beyond the standard SSL3 and TLS
try
{
result = webClient.UploadString(DestinationURI, stringToSend);
}
catch (WebException ex)
{
Dts.Events.FireError(0, "", "UnableToSendData: " + ex.Message.ToString() + ex.StackTrace, string.Empty, 0);
return;
}
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Below is a sample of 2 records for reference (\n is the start and ending of each data row), but of course we usually send about 9,000 records daily;
"123456|123456|TestUser1|Adam|Mphil/PhD History (PT) Year|3|Department of History||P||||XK|Information not sought|||123456|[email protected]|||||V1ZR-Y||HISs|H|M|1998-05-21||PGR|||0777XXXXXX|||2021-09-19|2022-09-10| |Test NOK: 58 Wiltshire Ave, HA8 5DG: 0208XXXXXX|||||\n678910|678910|TestUser2|Smith|Mphil/PhD Near and Middle East (PT) Year|12|Other Middle East||P||||XK|No known disability|||678910|[email protected]|||||T6ZT/D-Y||OMEs|H|M|1990-03-15||PGR|||079XXXXXXXX|||2022-09-26|2023-09-25| |Test NOK2: 45 Britannia Street, MK11 2ZQ: 0207XXXXXX|||||\n"
I must add, when we send the Post Request to the Web API, we log the response to table in SSMS - the table simple logs the request response, and in this case it's a '400 Bad Request' response. Is there any way to identify where exactly the Request falls over in a case where thousands of records are being sent, as in, can the offending records be identified in a simple way? I'm thinking maybe attempting to send the data one-by-one and if the Post Request encounters a record with bad data, it can be flagged.
Upvotes: 0
Views: 55