HttpWebRequest from WCF Service fails with - The request was aborted: Could not create SSL/TLS secure channel

Question

i read all the relevant threads on the error "The request was aborted: Could not create SSL/TLS secure channel" but none was applicable to my case.

I have a set of code that is executing against the Splunk HTTP Event collector REST API. if i execute the exact same code from a console C# app locally on my box, all works great.

When i execute it inside a WCF service it fails with the error above. Since the certificate Splunk Light Cloud service uses is a self-signed cert, i set up the following prior to executing my code.

ServicePointManager.Expect100Continue = true;
ServicePointManager.CheckCertificateRevocationList = false;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = new
RemoteCertificateValidationCallback(
     delegate { return true; }
     );

I also played around with the different options under ServicePointManager.SecurityProtocol, choosing SecurityProtocolType.Tls12 as well with no luck.

The error seems to be always the same. I enabled System.Net diagnostic tracing, but that did not seem helpful. I trimmed it to fit the post limit on stackoverflow.

System.Net Information: 0 : [3768] TlsStream#64025267::.ctor(host=input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, #certs=0)
System.Net Information: 0 : [3712] ConnectStream#21257040 - Sending headers
{
Accept: application/json, application/xml
Content-Type: application/json
Host: hqt-harp03:62677
Content-Length: 1231
Expect: 100-continue
}.
System.Net.Sockets Verbose: 0 : [3712] Socket#7894961::UnsafeBeginSend()
System.Net Information: 0 : [3768] Associating HttpWebRequest#6651366 with ConnectStream#48769823
System.Net.Sockets Verbose: 0 : [3712] Exiting Socket#7894961::UnsafeBeginSend()    -> OverlappedAsyncResult#34962020
System.Net.Sockets Verbose: 0 : [4740] Data from Socket#7894961::PostCompletion
System.Net.Sockets Verbose: 0 : [4740] 00000000 : 50 4F 53 54 20 2F 62 6F-6F 74 73 74 72 61 70 2D : POST /bootstrap-
System.Net.Sockets Verbose: 0 : [4740] 00000010 : 61 70 69 2F 6C 6F 67 20-48 54 54 50 2F 31 2E 31 : api/log HTTP/1.1
System.Net.Sockets Verbose: 0 : [4740] 00000020 : 0D 0A 41 63 63 65 70 74-3A 20 61 70 70 6C 69 63 : ..Accept: applic
System.Net.Sockets Verbose: 0 : [4740] 00000030 : 61 74 69 6F 6E 2F 6A 73-6F 6E 2C 20 61 70 70 6C : ation/json, appl
System.Net.Sockets Verbose: 0 : [4740] 00000040 : 69 63 61 74 69 6F 6E 2F-78 6D 6C 0D 0A 43 6F 6E : ication/xml..Con
System.Net.Sockets Verbose: 0 : [4740] 00000050 : 74 65 6E 74 2D 54 79 70-65 3A 20 61 70 70 6C 69 : tent-Type: appli
System.Net Information: 0 : [3768] HttpWebRequest#6651366 - Request: POST /services/collector/event HTTP/1.1

System.Net Information: 0 : [3768] ConnectStream#48769823 - Sending headers
{
Authorization: Splunk <removed for security>
Content-Type: application/json
Host: input-prd-p-kqcmtgqgwbzx.cloud.splunk.com:8088
Content-Length: 41
Expect: 100-continue
Connection: Close
}.
System.Net Information: 0 : [3768] SecureChannel#30995104::.ctor(hostname=input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, #clientCertificates=0, encryptionPolicy=RequireEncryption)
System.Net.Sockets Verbose: 0 : [4740] 000000B0 : 0A 0D 0A                                        : ...
System.Net.Sockets Verbose: 0 : [4740] Socket#7894961::EndSend(OverlappedAsyncResult#34962020)
System.Net.Sockets Verbose: 0 : [4740] Exiting Socket#7894961::EndSend()    -> Int32#179
System.Net.Sockets Verbose: 0 : [4740] Socket#7894961::UnsafeBeginReceive()
System.Net.Sockets Verbose: 0 : [4740] Exiting Socket#7894961::UnsafeBeginReceive()     -> OverlappedAsyncResult#44307732
System.Net.Sockets Verbose: 0 : [3712] Data from Socket#7894961::PostCompletion
System.Net.Sockets Verbose: 0 : [3712] 00000000 : 48 54 54 50 2F 31 2E 31-20 31 30 30 20 43 6F 6E : HTTP/1.1 100 Con
System.Net.Sockets Verbose: 0 : [3712] 00000010 : 74 69 6E 75 65 0D 0A 0D-0A                      : tinue....
System.Net.Sockets Verbose: 0 : [3712] Socket#7894961::EndReceive(OverlappedAsyncResult#44307732)
System.Net.Sockets Verbose: 0 : [3712] Exiting Socket#7894961::EndReceive()     -> Int32#25
System.Net Information: 0 : [3768] Enumerating security packages:
System.Net Information: 0 : [3712] Connection#45485984 - Received status line: Version=1.1, StatusCode=100, StatusDescription=Continue.
System.Net Information: 0 : [3712] Connection#45485984 - Received headers
{

}.
System.Net Information: 0 : [3768]     Negotiate
System.Net Information: 0 : [3768]     NegoExtender
System.Net Information: 0 : [3768]     Kerberos
System.Net Information: 0 : [3768]     NTLM
System.Net Information: 0 : [3768]     TSSSP
System.Net Information: 0 : [3768]     pku2u
System.Net Information: 0 : [3768]     WDigest
System.Net Information: 0 : [3768]     Schannel
System.Net Information: 0 : [3768]     Microsoft Unified Security Protocol Provider
System.Net Information: 0 : [3768]     CREDSSP
System.Net.Sockets Verbose: 0 : [3712] Socket#7894961::UnsafeBeginReceive()
System.Net Information: 0 : [3768] SecureChannel#30995104 - Left with 0 client certificates to choose from.
System.Net.Sockets Verbose: 0 : [3712] Exiting Socket#7894961::UnsafeBeginReceive()     -> OverlappedAsyncResult#35417413
System.Net Verbose: 0 : [3712] HttpWebRequest#41728762::EndGetRequestStream()
System.Net Verbose: 0 : [3712] Exiting HttpWebRequest#41728762::EndGetRequestStream()   -> ConnectStream#21257040
System.Net Verbose: 0 : [3712] ConnectStream#21257040::BeginWrite()
System.Net Verbose: 0 : [3712] Data from ConnectStream#21257040::BeginWrite
System.Net Information: 0 : [3768] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
System.Net Verbose: 0 : [3712] (printing 1024 out of 1231)
System.Net Verbose: 0 : [3712] 00000000 : 5B 7B 22 69 64 22 3A 22-30 30 30 30 30 30 30 30 : [{"id":"00000000
System.Net Verbose: 0 : [3712] 00000010 : 2D 30 30 30 30 2D 30 30-30 30 2D 30 30 30 30 2D : -0000-0000-0000-
System.Net Verbose: 0 : [3712] 00000020 : 30 30 30 30 30 30 30 30-30 30 30 30 22 2C 22 74 : 000000000000","t
System.Net Verbose: 0 : [3712] 00000030 : 65 78 74 22 3A 22 30 44-41 35 34 41 46 36 2D 30 : ext":"0DA54AF6-0
System.Net Verbose: 0 : [3712] 00000040 : 37 36 33 2D 34 45 41 35-2D 41 37 43 35 2D 37 33 : 763-4EA5-A7C5-73
System.Net Verbose: 0 : [3712] 00000050 : 35 45 43 33 44 39 34 36-42 38 22 2C 22 74 61 67 : 5EC3D946B8","tag
System.Net Verbose: 0 : [3712] 00000060 : 22 3A 22 4C 6F 67 41 67-67 72 65 67 61 74 6F 72 : ":"LogAggregator
System.Net Verbose: 0 : [3712] 00000070 : 2E 4C 6F 67 53 65 72 76-69 63 65 22 2C 22 6C 6F : .LogService","lo
System.Net Verbose: 0 : [3712] 00000080 : 67 4C 65 76 65 6C 22 3A-35 2C 22 6C 65 76 65 6C : gLevel":5,"level
System.Net Information: 0 : [3768] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Verbose: 0 : [3712] 00000090 : 22 3A 22 46 61 74 61 6C-22 2C 22 73 75 62 73 63 : ":"Fatal","subsc
System.Net Verbose: 0 : [3712] 000000A0 : 72 69 62 65 72 49 64 22-3A 22 30 30 30 30 30 30 : riberId":"000000
System.Net Verbose: 0 : [3712] 000000B0 : 30 30 2D 30 30 30 30 2D-30 30 30 30 2D 30 30 30 : 00-0000-0000-000
System.Net Verbose: 0 : [3712] 000000C0 : 30 2D 30 30 30 30 30 30-30 30 30 30 30 30 22 2C : 0-000000000000",
System.Net Verbose: 0 : [3712] 000000D0 : 22 63 75 73 74 6F 6D 65-72 49 64 22 3A 22 30 38 : "customerId":"08
System.Net Verbose: 0 : [3712] 000000E0 : 38 30 31 39 65 66 2D 33-31 66 65 2D 34 39 31 62 : 8019ef-31fe-491b
System.Net Verbose: 0 : [3712] 000000F0 : 2D 61 62 63 34 2D 31 34-37 36 32 30 64 36 37 38 : -abc4-147620d678
System.Net Verbose: 0 : [3712] 00000100 : 66 64 22 2C 22 76 65 72-73 69 6F 6E 49 64 22 3A : fd","versionId":
System.Net Verbose: 0 : [3712] 00000110 : 22 61 34 32 63 31 36 38-65 2D 37 38 65 64 2D 65 : "a42c168e-78ed-e
System.Net Verbose: 0 : [3712] 00000120 : 36 31 31 2D 38 31 31 38-2D 30 30 35 30 35 36 62 : 611-8118-005056b
System.Net Verbose: 0 : [3712] 00000130 : 34 33 64 39 30 22 2C 22-70 72 6F 76 69 64 65 72 : 43d90","provider
System.Net Information: 0 : [3768] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=119, returned code=ContinueNeeded).
System.Net Verbose: 0 : [3712] 00000140 : 49 64 22 3A 22 30 38 38-30 31 39 65 66 2D 33 31 : Id":"088019ef-31
System.Net Verbose: 0 : [3712] 00000150 : 66 65 2D 34 39 31 62 2D-61 62 63 34 2D 31 34 37 : fe-491b-abc4-147
System.Net.Sockets Verbose: 0 : [3768] Socket#20398441::Send()
System.Net Verbose: 0 : [3712] 00000160 : 36 32 30 64 36 37 38 66-64 22 2C 22 73 65 73 73 : 620d678fd","sess
System.Net Verbose: 0 : [3712] 00000170 : 69 6F 6E 49 64 22 3A 22-30 30 30 30 30 30 30 30 : ionId":"00000000
System.Net.Sockets Verbose: 0 : [3768] Data from Socket#20398441::Send
System.Net Verbose: 0 : [3712] 00000180 : 2D 30 30 30 30 2D 30 30-30 30 2D 30 30 30 30 2D : -0000-0000-0000-
System.Net Verbose: 0 : [3712] 00000190 : 30 30 30 30 30 30 30 30-30 30 30 30 22 2C 22 69 : 000000000000","i
System.Net Verbose: 0 : [3712] 000001A0 : 6E 73 74 61 6E 63 65 49-64 22 3A 22 37 39 30 61 : nstanceId":"790a
System.Net Verbose: 0 : [3712] 000001B0 : 61 63 33 36 2D 36 65 64-34 2D 34 64 39 61 2D 38 : ac36-6ed4-4d9a-8
System.Net Verbose: 0 : [3712] 000001C0 : 33 31 38 2D 33 37 31 31-64 39 63 30 30 38 33 61 : 318-3711d9c0083a
System.Net.Sockets Verbose: 0 : [3768] 00000000 : 16 03 01 00 72 01 00 00-6E 03 01 58 9B 92 5A DA : ....r...n..X..Z.
System.Net.Sockets Verbose: 0 : [3768] 00000010 : B5 7D 61 E1 FA 9C 47 91-9B 1C A1 D5 49 9E B8 0E : .}a...G.....I...
System.Net.Sockets Verbose: 0 : [3768] 00000020 : 53 9B F9 E4 30 7A CB C7-4D 44 52 00 00 06 00 2F : S...0z..MDR..../
System.Net.Sockets Verbose: 0 : [3768] 00000030 : 00 35 00 0A 01 00 00 3F-00 00 00 2E 00 2C 00 00 : .5.....?.....,..
System.Net.Sockets Verbose: 0 : [3768] 00000040 : 29 69 6E 70 75 74 2D 70-72 64 2D 70 2D 6B 71 63 : )input-prd-p-kqc
System.Net Verbose: 0 : [3712] 000001D0 : 22 2C 22 6C 6F 63 61 6C-54 72 61 6E 73 61 63 74 : ","localTransact
System.Net Verbose: 0 : [3712] 000001E0 : 69 6F 6E 49 64 22 3A 6E-75 6C 6C 2C 22 64 69 73 : ionId":null,"dis
System.Net Verbose: 0 : [3712] 000001F0 : 74 72 69 62 75 74 65 64-54 72 61 6E 73 61 63 74 : tributedTransact
System.Net Verbose: 0 : [3712] 00000200 : 69 6F 6E 49 64 22 3A 6E-75 6C 6C 2C 22 6D 61 63 : ionId":null,"mac
System.Net Verbose: 0 : [3712] 00000210 : 68 69 6E 65 4E 61 6D 65-22 3A 22 48 51 54 2D 48 : hineName":"HQT-H
System.Net Verbose: 0 : [3712] 00000220 : 41 52 50 30 33 22 2C 22-74 69 6D 65 73 74 61 6D : ARP03","timestam
System.Net Verbose: 0 : [3712] 00000230 : 70 22 3A 22 32 30 31 37-2D 30 32 2D 30 38 54 32 : p":"2017-02-08T2
System.Net.Sockets Verbose: 0 : [3768] 00000050 : 6D 74 67 71 67 77 62 7A-78 2E 63 6C 6F 75 64 2E : mtgqgwbzx.cloud.
System.Net.Sockets Verbose: 0 : [3768] 00000060 : 73 70 6C 75 6E 6B 2E 63-6F 6D 00 23 00 00 00 17 : splunk.com.#....
System.Net.Sockets Verbose: 0 : [3768] 00000070 : 00 00 FF 01 00 01 00                            : .......
System.Net.Sockets Verbose: 0 : [3768] Exiting Socket#20398441::Send()  -> Int32#119
System.Net.Sockets Verbose: 0 : [3768] Socket#20398441::Receive()
System.Net Verbose: 0 : [3712] 00000240 : 31 3A 34 39 3A 31 34 2E-30 39 35 32 39 35 5A 22 : 1:49:14.095295Z"
System.Net Verbose: 0 : [3712] 00000250 : 7D 2C 7B 22 69 64 22 3A-22 30 30 30 30 30 30 30 : },{"id":"0000000
System.Net Verbose: 0 : [3712] 00000260 : 30 2D 30 30 30 30 2D 30-30 30 30 2D 30 30 30 30 : 0-0000-0000-0000
System.Net Verbose: 0 : [3712] 00000270 : 2D 30 30 30 30 30 30 30-30 30 30 30 30 22 2C 22 : -000000000000","
System.Net Verbose: 0 : [3712] 00000280 : 74 65 78 74 22 3A 22 68-74 74 70 73 3A 2F 2F 69 : text":"https://i
System.Net Verbose: 0 : [3712] 00000290 : 6E 70 75 74 2D 70 72 64-2D 70 2D 6B 71 63 6D 74 : nput-prd-p-kqcmt
System.Net Verbose: 0 : [3712] 000002A0 : 67 71 67 77 62 7A 78 2E-63 6C 6F 75 64 2E 73 70 : gqgwbzx.cloud.sp
System.Net Verbose: 0 : [3712] 000002B0 : 6C 75 6E 6B 2E 63 6F 6D-3A 38 30 38 38 2F 73 65 : lunk.com:8088/se
System.Net Verbose: 0 : [3712] 000002C0 : 72 76 69 63 65 73 2F 63-6F 6C 6C 65 63 74 6F 72 : rvices/collector
System.Net Verbose: 0 : [3712] 000002D0 : 2F 65 76 65 6E 74 22 2C-22 74 61 67 22 3A 22 4C : /event","tag":"L
System.Net Verbose: 0 : [3712] 000002E0 : 6F 67 41 67 67 72 65 67-61 74 6F 72 2E 4C 6F 67 : ogAggregator.Log
System.Net Verbose: 0 : [3712] 000002F0 : 53 65 72 76 69 63 65 22-2C 22 6C 6F 67 4C 65 76 : Service","logLev
System.Net Verbose: 0 : [3712] 00000300 : 65 6C 22 3A 35 2C 22 6C-65 76 65 6C 22 3A 22 46 : el":5,"level":"F
System.Net Verbose: 0 : [3712] 00000310 : 61 74 61 6C 22 2C 22 73-75 62 73 63 72 69 62 65 : atal","subscribe
System.Net Verbose: 0 : [3712] 00000320 : 72 49 64 22 3A 22 30 30-30 30 30 30 30 30 2D 30 : rId":"00000000-0
System.Net Verbose: 0 : [3712] 00000330 : 30 30 30 2D 30 30 30 30-2D 30 30 30 30 2D 30 30 : 000-0000-0000-00
System.Net Verbose: 0 : [3712] 00000340 : 30 30 30 30 30 30 30 30-30 30 22 2C 22 63 75 73 : 0000000000","cus
System.Net Verbose: 0 : [3712] 00000350 : 74 6F 6D 65 72 49 64 22-3A 22 30 38 38 30 31 39 : tomerId":"088019
System.Net Verbose: 0 : [3712] 00000360 : 65 66 2D 33 31 66 65 2D-34 39 31 62 2D 61 62 63 : ef-31fe-491b-abc
System.Net Verbose: 0 : [3712] 00000370 : 34 2D 31 34 37 36 32 30-64 36 37 38 66 64 22 2C : 4-147620d678fd",
System.Net Verbose: 0 : [3712] 00000380 : 22 76 65 72 73 69 6F 6E-49 64 22 3A 22 61 34 32 : "versionId":"a42
System.Net Verbose: 0 : [3712] 00000390 : 63 31 36 38 65 2D 37 38-65 64 2D 65 36 31 31 2D : c168e-78ed-e611-
System.Net Verbose: 0 : [3712] 000003A0 : 38 31 31 38 2D 30 30 35-30 35 36 62 34 33 64 39 : 8118-005056b43d9
System.Net Verbose: 0 : [3712] 000003B0 : 30 22 2C 22 70 72 6F 76-69 64 65 72 49 64 22 3A : 0","providerId":
System.Net Verbose: 0 : [3712] 000003C0 : 22 30 38 38 30 31 39 65-66 2D 33 31 66 65 2D 34 : "088019ef-31fe-4
System.Net Verbose: 0 : [3712] 000003D0 : 39 31 62 2D 61 62 63 34-2D 31 34 37 36 32 30 64 : 91b-abc4-147620d
System.Net Verbose: 0 : [3712] 000003E0 : 36 37 38 66 64 22 2C 22-73 65 73 73 69 6F 6E 49 : 678fd","sessionI
System.Net Verbose: 0 : [3712] 000003F0 : 64 22 3A 22 30 30 30 30-30 30 30 30 2D 30 30 30 : d":"00000000-000
System.Net.Sockets Verbose: 0 : [3712] Socket#7894961::BeginSend()
System.Net.Sockets Verbose: 0 : [3712] Exiting Socket#7894961::BeginSend()  -> OverlappedAsyncResult#57731671
System.Net.Sockets Verbose: 0 : [4740] Data from Socket#7894961::PostCompletion
System.Net.Sockets Verbose: 0 : [4740] (printing 1024 out of 1231)
System.Net.Sockets Verbose: 0 : [4740] 00000000 : 5B 7B 22 69 64 22 3A 22-30 30 30 30 30 30 30 30 : [{"id":"00000000
System.Net.Sockets Verbose: 0 : [4740] 00000010 : 2D 30 30 30 30 2D 30 30-30 30 2D 30 30 30 30 2D : -0000-0000-0000-
System.Net.Sockets Verbose: 0 : [4740] 00000020 : 30 30 30 30 30 30 30 30-30 30 30 30 22 2C 22 74 : 000000000000","t
System.Net.Sockets Verbose: 0 : [4740] 00000030 : 65 78 74 22 3A 22 30 44-41 35 34 41 46 36 2D 30 : ext":"0DA54AF6-0
System.Net Verbose: 0 : [3712] Exiting ConnectStream#21257040::BeginWrite()     -> NestedSingleAsyncResult#10279613
System.Net.Sockets Verbose: 0 : [4740] 00000040 : 37 36 33 2D 34 45 41 35-2D 41 37 43 35 2D 37 33 : 763-4EA5-A7C5-73
System.Net.Sockets Verbose: 0 : [4740] 00000050 : 35 45 43 33 44 39 34 36-42 38 22 2C 22 74 61 67 : 5EC3D946B8","tag
System.Net.Sockets Verbose: 0 : [4740] 00000060 : 22 3A 22 4C 6F 67 41 67-67 72 65 67 61 74 6F 72 : ":"LogAggregator
System.Net.Sockets Verbose: 0 : [4740] 00000070 : 2E 4C 6F 67 53 65 72 76-69 63 65 22 2C 22 6C 6F : .LogService","lo
System.Net.Sockets Verbose: 0 : [4740] 00000080 : 67 4C 65 76 65 6C 22 3A-35 2C 22 6C 65 76 65 6C : gLevel":5,"level
System.Net.Sockets Verbose: 0 : [4740] 00000090 : 22 3A 22 46 61 74 61 6C-22 2C 22 73 75 62 73 63 : ":"Fatal","subsc
System.Net.Sockets Verbose: 0 : [4740] 000000A0 : 72 69 62 65 72 49 64 22-3A 22 30 30 30 30 30 30 : riberId":"000000
System.Net.Sockets Verbose: 0 : [4740] 000000B0 : 30 30 2D 30 30 30 30 2D-30 30 30 30 2D 30 30 30 : 00-0000-0000-000
System.Net.Sockets Verbose: 0 : [4740] 000000C0 : 30 2D 30 30 30 30 30 30-30 30 30 30 30 30 22 2C : 0-000000000000",
System.Net.Sockets Verbose: 0 : [4740] 000000D0 : 22 63 75 73 74 6F 6D 65-72 49 64 22 3A 22 30 38 : "customerId":"08
System.Net.Sockets Verbose: 0 : [4740] 000000E0 : 38 30 31 39 65 66 2D 33-31 66 65 2D 34 39 31 62 : 8019ef-31fe-491b
System.Net.Sockets Verbose: 0 : [4740] 000000F0 : 2D 61 62 63 34 2D 31 34-37 36 32 30 64 36 37 38 : -abc4-147620d678
System.Net.Sockets Verbose: 0 : [4740] 00000100 : 66 64 22 2C 22 76 65 72-73 69 6F 6E 49 64 22 3A : fd","versionId":
System.Net.Sockets Verbose: 0 : [4740] 00000110 : 22 61 34 32 63 31 36 38-65 2D 37 38 65 64 2D 65 : "a42c168e-78ed-e
System.Net.Sockets Verbose: 0 : [3712] Data from Socket#7894961::PostCompletion
System.Net.Sockets Verbose: 0 : [3712] 00000000 : 48 54 54 50 2F 31 2E 31-20 32 30 31 20 43 72 65 : HTTP/1.1 201 Cre
System.Net.Sockets Verbose: 0 : [3712] 00000010 : 61 74 65 64 0D 0A 43 6F-6E 74 65 6E 74 2D 4C 65 : ated..Content-Le
System.Net.Sockets Verbose: 0 : [3712] 00000020 : 6E 67 74 68 3A 20 30 0D-0A 53 65 72 76 65 72 3A : ngth: 0..Server:
System.Net.Sockets Verbose: 0 : [3712] 00000030 : 20 4D 69 63 72 6F 73 6F-66 74 2D 48 54 54 50 41 :  Microsoft-HTTPA
System.Net.Sockets Verbose: 0 : [3712] 00000040 : 50 49 2F 32 2E 30 0D 0A-44 61 74 65 3A 20 57 65 : PI/2.0..Date: We
System.Net.Sockets Verbose: 0 : [3712] 00000050 : 64 2C 20 30 38 20 46 65-62 20 32 30 31 37 20 32 : d, 08 Feb 2017 2
System.Net.Sockets Verbose: 0 : [3712] 00000060 : 31 3A 34 39 3A 31 33 20-47 4D 54 0D 0A 0D 0A    : 1:49:13 GMT....
System.Net.Sockets Verbose: 0 : [3712] Socket#7894961::EndReceive(OverlappedAsyncResult#35417413)
System.Net.Sockets Verbose: 0 : [3712] Exiting Socket#7894961::EndReceive()     -> Int32#111
System.Net Information: 0 : [3712] Connection#45485984 - Received status line: Version=1.1, StatusCode=201, StatusDescription=Created.
System.Net Information: 0 : [3712] Connection#45485984 - Received headers
{
Content-Length: 0
Date: Wed, 08 Feb 2017 21:49:13 GMT
Server: Microsoft-HTTPAPI/2.0
}.
System.Net Information: 0 : [3712] ConnectStream#33939037::ConnectStream(Buffered 0 bytes.)
System.Net.Sockets Verbose: 0 : [4740] 00000120 : 36 31 31 2D 38 31 31 38-2D 30 30 35 30 35 36 62 : 611-8118-005056b
System.Net.Sockets Verbose: 0 : [4740] 00000130 : 34 33 64 39 30 22 2C 22-70 72 6F 76 69 64 65 72 : 43d90","provider
System.Net.Sockets Verbose: 0 : [4740] 00000140 : 49 64 22 3A 22 30 38 38-30 31 39 65 66 2D 33 31 : Id":"088019ef-31
System.Net.Sockets Verbose: 0 : [4740] 00000150 : 66 65 2D 34 39 31 62 2D-61 62 63 34 2D 31 34 37 : fe-491b-abc4-147
System.Net.Sockets Verbose: 0 : [4740] 00000160 : 36 32 30 64 36 37 38 66-64 22 2C 22 73 65 73 73 : 620d678fd","sess
System.Net.Sockets Verbose: 0 : [4740] 00000170 : 69 6F 6E 49 64 22 3A 22-30 30 30 30 30 30 30 30 : ionId":"00000000
System.Net.Sockets Verbose: 0 : [4740] 00000180 : 2D 30 30 30 30 2D 30 30-30 30 2D 30 30 30 30 2D : -0000-0000-0000-
System.Net.Sockets Verbose: 0 : [4740] 00000190 : 30 30 30 30 30 30 30 30-30 30 30 30 22 2C 22 69 : 000000000000","i
System.Net.Sockets Verbose: 0 : [4740] 000001A0 : 6E 73 74 61 6E 63 65 49-64 22 3A 22 37 39 30 61 : nstanceId":"790a
System.Net.Sockets Verbose: 0 : [4740] 000001B0 : 61 63 33 36 2D 36 65 64-34 2D 34 64 39 61 2D 38 : ac36-6ed4-4d9a-8
System.Net.Sockets Verbose: 0 : [4740] 000001C0 : 33 31 38 2D 33 37 31 31-64 39 63 30 30 38 33 61 : 318-3711d9c0083a
System.Net.Sockets Verbose: 0 : [4740] 000001D0 : 22 2C 22 6C 6F 63 61 6C-54 72 61 6E 73 61 63 74 : ","localTransact
System.Net.Sockets Verbose: 0 : [4740] 000001E0 : 69 6F 6E 49 64 22 3A 6E-75 6C 6C 2C 22 64 69 73 : ionId":null,"dis
System.Net.Sockets Verbose: 0 : [4740] 000001F0 : 74 72 69 62 75 74 65 64-54 72 61 6E 73 61 63 74 : tributedTransact
System.Net.Sockets Verbose: 0 : [4740] 00000200 : 69 6F 6E 49 64 22 3A 6E-75 6C 6C 2C 22 6D 61 63 : ionId":null,"mac
System.Net.Sockets Verbose: 0 : [4740] 00000210 : 68 69 6E 65 4E 61 6D 65-22 3A 22 48 51 54 2D 48 : hineName":"HQT-H
System.Net.Sockets Verbose: 0 : [4740] 00000220 : 41 52 50 30 33 22 2C 22-74 69 6D 65 73 74 61 6D : ARP03","timestam
System.Net.Sockets Verbose: 0 : [4740] 00000230 : 70 22 3A 22 32 30 31 37-2D 30 32 2D 30 38 54 32 : p":"2017-02-08T2
System.Net.Sockets Verbose: 0 : [4740] 00000240 : 31 3A 34 39 3A 31 34 2E-30 39 35 32 39 35 5A 22 : 1:49:14.095295Z"
System.Net.Sockets Verbose: 0 : [4740] 00000250 : 7D 2C 7B 22 69 64 22 3A-22 30 30 30 30 30 30 30 : },{"id":"0000000
System.Net.Sockets Verbose: 0 : [4740] 00000260 : 30 2D 30 30 30 30 2D 30-30 30 30 2D 30 30 30 30 : 0-0000-0000-0000
System.Net.Sockets Verbose: 0 : [4740] 00000270 : 2D 30 30 30 30 30 30 30-30 30 30 30 30 22 2C 22 : -000000000000","
System.Net.Sockets Verbose: 0 : [4740] 00000280 : 74 65 78 74 22 3A 22 68-74 74 70 73 3A 2F 2F 69 : text":"https://i
System.Net.Sockets Verbose: 0 : [4740] 00000290 : 6E 70 75 74 2D 70 72 64-2D 70 2D 6B 71 63 6D 74 : nput-prd-p-kqcmt
System.Net.Sockets Verbose: 0 : [4740] 000002A0 : 67 71 67 77 62 7A 78 2E-63 6C 6F 75 64 2E 73 70 : gqgwbzx.cloud.sp
System.Net.Sockets Verbose: 0 : [4740] 000002B0 : 6C 75 6E 6B 2E 63 6F 6D-3A 38 30 38 38 2F 73 65 : lunk.com:8088/se
System.Net.Sockets Verbose: 0 : [4740] 000002C0 : 72 76 69 63 65 73 2F 63-6F 6C 6C 65 63 74 6F 72 : rvices/collector
System.Net.Sockets Verbose: 0 : [4740] 000002D0 : 2F 65 76 65 6E 74 22 2C-22 74 61 67 22 3A 22 4C : /event","tag":"L
System.Net.Sockets Verbose: 0 : [4740] 000002E0 : 6F 67 41 67 67 72 65 67-61 74 6F 72 2E 4C 6F 67 : ogAggregator.Log
System.Net.Sockets Verbose: 0 : [4740] 000002F0 : 53 65 72 76 69 63 65 22-2C 22 6C 6F 67 4C 65 76 : Service","logLev
System.Net.Sockets Verbose: 0 : [4740] 00000300 : 65 6C 22 3A 35 2C 22 6C-65 76 65 6C 22 3A 22 46 : el":5,"level":"F
System.Net.Sockets Verbose: 0 : [4740] 00000310 : 61 74 61 6C 22 2C 22 73-75 62 73 63 72 69 62 65 : atal","subscribe
System.Net.Sockets Verbose: 0 : [4740] 00000320 : 72 49 64 22 3A 22 30 30-30 30 30 30 30 30 2D 30 : rId":"00000000-0
System.Net.Sockets Verbose: 0 : [4740] 00000330 : 30 30 30 2D 30 30 30 30-2D 30 30 30 30 2D 30 30 : 000-0000-0000-00
System.Net.Sockets Verbose: 0 : [4740] 00000340 : 30 30 30 30 30 30 30 30-30 30 22 2C 22 63 75 73 : 0000000000","cus
System.Net.Sockets Verbose: 0 : [4740] 00000350 : 74 6F 6D 65 72 49 64 22-3A 22 30 38 38 30 31 39 : tomerId":"088019
System.Net.Sockets Verbose: 0 : [4740] 00000360 : 65 66 2D 33 31 66 65 2D-34 39 31 62 2D 61 62 63 : ef-31fe-491b-abc
System.Net.Sockets Verbose: 0 : [4740] 00000370 : 34 2D 31 34 37 36 32 30-64 36 37 38 66 64 22 2C : 4-147620d678fd",
System.Net.Sockets Verbose: 0 : [4740] 00000380 : 22 76 65 72 73 69 6F 6E-49 64 22 3A 22 61 34 32 : "versionId":"a42
System.Net.Sockets Verbose: 0 : [4740] 00000390 : 63 31 36 38 65 2D 37 38-65 64 2D 65 36 31 31 2D : c168e-78ed-e611-
System.Net.Sockets Verbose: 0 : [4740] 000003A0 : 38 31 31 38 2D 30 30 35-30 35 36 62 34 33 64 39 : 8118-005056b43d9
System.Net.Sockets Verbose: 0 : [4740] 000003B0 : 30 22 2C 22 70 72 6F 76-69 64 65 72 49 64 22 3A : 0","providerId":
System.Net.Sockets Verbose: 0 : [4740] 000003C0 : 22 30 38 38 30 31 39 65-66 2D 33 31 66 65 2D 34 : "088019ef-31fe-4
System.Net.Sockets Verbose: 0 : [4740] 000003D0 : 39 31 62 2D 61 62 63 34-2D 31 34 37 36 32 30 64 : 91b-abc4-147620d
System.Net.Sockets Verbose: 0 : [4740] 000003E0 : 36 37 38 66 64 22 2C 22-73 65 73 73 69 6F 6E 49 : 678fd","sessionI
System.Net.Sockets Verbose: 0 : [4740] 000003F0 : 64 22 3A 22 30 30 30 30-30 30 30 30 2D 30 30 30 : d":"00000000-000
System.Net.Sockets Verbose: 0 : [4740] Socket#7894961::EndSend(OverlappedAsyncResult#57731671)
System.Net.Sockets Verbose: 0 : [4740] Exiting Socket#7894961::EndSend()    -> Int32#1231
System.Net Verbose: 0 : [4740] ConnectStream#21257040::EndWrite()
System.Net Verbose: 0 : [4740] Exiting ConnectStream#21257040::EndWrite() 
System.Net Verbose: 0 : [4740] ConnectStream#21257040::Close()
System.Net Verbose: 0 : [4740] Exiting ConnectStream#21257040::Close() 
System.Net Verbose: 0 : [4740] HttpWebRequest#41728762::BeginGetResponse()
System.Net Information: 0 : [4740] Associating HttpWebRequest#41728762 with ConnectStream#33939037
System.Net Information: 0 : [4740] Associating HttpWebRequest#41728762 with HttpWebResponse#2882782
System.Net Verbose: 0 : [4740] HttpWebRequest#41728762::EndGetResponse()
System.Net Verbose: 0 : [4740] Exiting HttpWebRequest#41728762::EndGetResponse()    -> HttpWebResponse#2882782
System.Net.Http Verbose: 0 : [4740] HttpResponseMessage#11194425::.ctor(StatusCode: 201, ReasonPhrase: '')
System.Net.Http Verbose: 0 : [4740] Exiting HttpResponseMessage#11194425::.ctor() 
System.Net.Http Information: 0 : [4740] Associating HttpResponseMessage#11194425 with HttpRequestMessage#13009416
System.Net Verbose: 0 : [4740] HttpWebResponse#2882782::GetResponseStream()
System.Net Information: 0 : [4740] ContentLength=0
System.Net Verbose: 0 : [4740] Exiting HttpWebResponse#2882782::GetResponseStream()     -> ConnectStream#33939037
System.Net.Http Verbose: 0 : [4740] StreamContent#12117429::.ctor()
System.Net.Http Verbose: 0 : [4740] Exiting StreamContent#12117429::.ctor() 
System.Net.Http Information: 0 : [4740] Associating StreamContent#12117429 with WebExceptionWrapperStream#34540003
System.Net.Http Information: 0 : [4740] Associating HttpResponseMessage#11194425 with StreamContent#12117429
System.Net Verbose: 0 : [4740] ConnectStream#33939037::BeginRead()
System.Net Verbose: 0 : [4740] Exiting ConnectStream#33939037::BeginRead()  -> NestedSingleAsyncResult#62046035
System.Net Verbose: 0 : [4740] ConnectStream#33939037::EndRead()
System.Net Verbose: 0 : [4740] Exiting ConnectStream#33939037::EndRead()    -> Int32#0
System.Net Verbose: 0 : [4740] ConnectStream#33939037::Close()
System.Net Verbose: 0 : [4740] Exiting ConnectStream#33939037::Close() 
System.Net.Http Information: 0 : [4740] HttpClient#2808346 - Request for HttpRequestMessage#13009416 completed successfully. Returning response HttpResponseMessage#11194425: StatusCode: 201, ReasonPhrase: 'Created', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Wed, 08 Feb 2017 21:49:13 GMT
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 0
}
System.Net Verbose: 0 : [4740] Exiting HttpWebRequest#41728762::BeginGetResponse()  -> ContextAwareResult#45776532
System.Net.Sockets Verbose: 0 : [11656] Socket#7894961::Dispose()
System.Net.Sockets Verbose: 0 : [3768] Data from Socket#20398441::Receive
System.Net.Sockets Verbose: 0 : [3768] 00000000 : 15 03 01 00 02                                  : .....
System.Net.Sockets Verbose: 0 : [3768] Exiting Socket#20398441::Receive()   -> Int32#5
System.Net.Sockets Verbose: 0 : [3768] Socket#20398441::Receive()
System.Net.Sockets Verbose: 0 : [3768] Data from Socket#20398441::Receive
System.Net.Sockets Verbose: 0 : [3768] 00000005 : 02 28                                           : .(
System.Net.Sockets Verbose: 0 : [3768] Exiting Socket#20398441::Receive()   -> Int32#2
System.Net Information: 0 : [3768] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 1b3a3e40:542be2e580, targetName = input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [3768] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=IllegalMessage).
System.Net.Sockets Verbose: 0 : [3768] Socket#20398441::Dispose()
System.Net Error: 0 : [3768] Exception in HttpWebRequest#6651366:: - The request was aborted: Could not create SSL/TLS secure channel..
System.Net Error: 0 : [3768] Exception in HttpWebRequest#6651366::EndGetRequestStream - The request was aborted: Could not create SSL/TLS secure channel..

It is worth noting that the main different in an SSL trace between a successful and the failed one above is the following:

Failed Code:

InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=IllegalMessage).

Successful Code:

System.Net Information: 0 : [17520] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).

Any idea what's special about WCF services that's failing to make an HttWebRequest to an https-based RESTful service?

thanks in advance!

Answer 1

I was finally able to solve this and updating this page for future folks. In trying to understand the issue, i concentrated my investigation in 3 areas that could possibly cause the response of an IllegalMessage (Win32 SChannel Native Error Code: 0x80090326)

1. Did the Splunk site use ECC (elliptic curves cryptography) in their certificate and my server could not support it
2. Was SSL renegotiation not working
3. Did the ciphers my computer was producing not matching the certificate expectations

wireshark and microsoft message analyzer helped a little bit, but since I did not have the private key of the Splunk certificate, i could not decrypt a lot of the content. Fiddler, with the man in the middle proxy, gave me some useful data as well.

Eventually, #1 was not an issue. #2 was an issue and I discovered https://support.microsoft.com/en-us/help/2745582/.net-framework-4.5-hotfix-resolves-httpwebrequest-and-sslstream-issues. With that in mind, I updated my computers to .NET 4.6.2

Once I got past all the other issues, #3 became my problem. According to this article, https://technet.microsoft.com/en-us/library/dn786419(v=ws.11).aspx, Windows Server 2012 R2 and Win2k16 support the same TLS and cipher suits. After lots of debugging, i discovered our domain policy had a group policy that restricted the ciphers available for use and demanded a specific order depending on the Operating System you were using. After manual trial and error, I was able to pinpoint the exact 3 ciphers needed by the Splunk Light public endpoint for HTTP event collection:

TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384

the fact that was i executing the code from a WCF service ended up playing no role in this.