DLL causing web app to hang indefinitely, no error

Question

I apologize if this has been addressed. I've read through several similar questions, but have been unable to find anything quite like the situation I'm experiencing, or is for a specific technology that is not being utilized in this case.

I have a web app (HTML, JavaScript, VB.NET on the backend). This app utilizes an in-house DLL that was written before I started here. That in-house DLL also utilizes a 3rd party DLL. The 3rd party DLL is supposed to perform several functions related to handling .dbf database files.

When I have everything set up locally (Visual Studio 2013, IIS Express) everything works flawlessly. I can input my criteria, my code does what it's supposed to do, eventually calling our in-house DLL, that DLL goes through its routines, eventually calling the 3rd party DLL, results are returned and displayed in the browser, just as one would expect. However, when I set it all up on our web server in IIS, something goes wrong.

I now have the app set up on Server 2012 R2 with IIS 8.5. I have duplicated the directory structure on my local machine and have debugged to ensure the correct paths are being used and have checked file permissions. Regardless, when I fire up the web app, I make my selections, but the app gets stuck on the wait screen.

Though I cannot actually debug on the web server, I have been able to track the problem to when the 3rd party DLL is called by placing a "MsgBox("Test")" along the code path until it stopped throwing the error about displaying a server side message in client side code (or something similar). The errors stop at the point of the 3rd party DLL call. I've run several Fiddler traces, and the call to my function that eventually gets to the 3rd party DLL never receives a response.

Here are the properties for that call:

To me it appears as if the server is never responding with the data from the DLL. Add to that the fact that if I go to the webserver and attempt to delete the directory of the application, it tells me that the 3rd party DLL cannot be deleted because it is in-use by w3wp.exe.

All this seems to point to the fact that the server is accessing the 3rd party DLL, but hanging up somewhere inside. I know it would be difficult to pinpoint the exact issue without knowing the setup of our webserver or the code itself, but does anyone know of something I might check? Permissions seem ok and there are no errors that I can find in the IIS logs or in the Windows System logs. I would think maybe firewall, but I'm trying to run it from the browser on the webserver itself. Is there some way to allow any messages that may appear inside the DLL to come through, or some tool that could capture in more detail what is happening? Is this more likely to be a configuration issue, or a code issue?

Any help would be greatly appreciated! Thanks in advance!

---

UPDATE: Found some info in the Fiddler Log that may be useful, but I'm not sure what to make of it. Maybe someone that knows Fiddler better than I will know what this means, if anything, with regard to the original issue.

-= Fiddler Event Log =-

16:06:25:8250 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. https://github.com/ssllabs/research/wiki/Long-Handshake-Intolerance
16:06:25:8280 !SecureClientPipeDirect failed: System.Security.Authentication.AuthenticationException A call to SSPI failed, see inner exception. < The certificate chain was issued by an authority that is not trusted on pipe to (CN=osce11.icrc.trendmicro.com, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com).
16:06:25:9110 HTTPSLint> Warning: ClientHello record was 508 bytes long. Some servers have problems with ClientHello's greater than 255 bytes. https://github.com/ssllabs/research/wiki/Long-Handshake-Intolerance
16:06:25:9110 !SecureClientPipeDirect failed: System.Security.Authentication.AuthenticationException A call to SSPI failed, see inner exception. < The certificate chain was issued by an authority that is not trusted on pipe to (CN=osce11.icrc.trendmicro.com, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com).
16:06:29:9998 [Fiddler] No HTTPS request was received from (chrome:6568) new client socket, port 61029.
16:06:37:1810 [Fiddler] No HTTP request was received from (chrome:6568) new client socket, port 61026.
16:06:37:1810 [Fiddler] No HTTP request was received from (chrome:6568) new client socket, port 61025.
16:06:37:1810 [Fiddler] No HTTP request was received from (chrome:6568) new client socket, port 61024.
16:06:37:1810 [Fiddler] No HTTP request was received from (chrome:6568) new client socket, port 61023.
16:06:37:1810 [Fiddler] No HTTP request was received from (chrome:6568) new client socket, port 61022.
16:06:38:3785 !SecureClientPipeDirect failed: System.Security.Authentication.AuthenticationException A call to SSPI failed, see inner exception. < An unknown error occurred while processing the certificate on pipe to (CN=osce110-en.fbs10.trendmicro.com, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com).

Answer 1

I believe I have figured out the issue, thanks to Robert's comments!

I installed the full version of IIS on my local machine, rather than IIS express and set it up just as it is on the webserver. A funny thing happened, I got the same results I was seeing on the server. Not that IIS itself helped, but it allowed me to set the Identity to ApplicationPoolIdentity, whereas I believe IIS Express must use my Windows credentials.

Anyway, I tried all the built in Identities for the App Pool, but none worked, so I set it to use my Windows domain credentials and it worked. From there I set the App Pool back to use ApplicationPoolIdentity and modified folder permissions until it worked.

Turns out, all I needed to do was give the Users group Modify permissions.

Thanks again!

Answer 2

OK, it's been more than a day with now answers, so I'll fall back on the "any help" clause - I don't know exactly what's happening, but maybe some of my "what I would try" might help you.

First, it does sound like a permission problem, though probably a policy as opposed to file permission. Is your IIS app pool using a domain account or a local service account? If it's running off a local service account, make sure your local computer is using the same service for its app pool, and that the Local Policy for that service account is set the same on your PC as on the server.

Where do files reside? You mentioned a DBF file, is that by any chance on a local drive for your PC and a network drive for the server? Sometimes that can cause a real headache, NTLM multi-hop, etc. Try moving your file to the same network drive if yours is local and the servers is network.

If you are using a domain account for the app pool on the server, use the same domain account for the app pool on your local machine, and again, make sure the policy grants the same permissions on your machine as it does on the server.

If that doesn't sort it out, I'd look in the in-house DLL and make a note of the API signatures of all of the calls made to the 3rd party DLL. I'd then make 2 stubs - one a harness that made the same calls to the 3rd party DLL (with some dummy values) and one that mimicked the 3rd party DLL but returned just some fixed dummy data, or maybe some dummy data with some diagnostic stuff like username embedded in it. I'd then both together to make sure you can make simple calls and each half works as expected. When that's all good, use your harness to test the 3rd party DLL on the server, does it run at all? If it does, try hooking your harness into a simple ASP page and see if you can exercise the 3rd party DLL with simple cases using the IIS app pool. Also try using your harness to work your stub on the server using the app pool, and verify what user context everything is running under.

It would also help to know a bit more about the authentication infrastructure you have - where are local service accounts used and where are domain accounts used, are you using NTLM, Delegated Constraints, Kerberos, impersonation, etc. Don't post details like account names, just DomainAcct1, DomainAcct2, etc.

And BTW, I've been in your shoes before and it sucks, you have my sympathy :-)