Ubuntu IoT Edge edgeHub module fails with error (139) Access to the path '/tmp/edgeHub/edgeHub' is denied

Question

I am deploying a production version of a working OPCPublisher module for IoT Edge, meaning I have a fully working solution in development.

However when I try to deploy the same thing in production server I get an error from edgeHub module.

The server is Ubuntu Server 18.04 and I used the official document to install edge: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-linux#uninstall-iot-edge

Now there were tons of problems in the installation process due to very tight access rights to internet, but all those were resolved, edge was installed successfully and then linked to the OPCPublisher module running in MS azure IoT hub.

At this point the edge on Linux downloaded the docker images for OPCPublisher, edgeAgent and edgeHub. Publihser and agent went through without a hitch (well after I battled with more access rights to docker image repos) but the edgeHub fails with code 139:

and if I look at the logs for the hub, they say:

Unhandled Exception: System.AggregateException: One or more errors occurred. (Access to the path '/tmp/edgeHub/edgeHub' is denied.) ---> System.UnauthorizedAccessException: Access to the path '/tmp/edgeHub/edgeHub' is denied. ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at Microsoft.Azure.Devices.Edge.Hub.Service.DependencyManager.GetStoragePath() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/DependencyManager.cs:line 282
   at Microsoft.Azure.Devices.Edge.Hub.Service.DependencyManager.GetStoreAndForwardConfiguration() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/DependencyManager.cs:line 245
   at Microsoft.Azure.Devices.Edge.Hub.Service.DependencyManager.Register(ContainerBuilder builder) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/DependencyManager.cs:line 79
   at Microsoft.Azure.Devices.Edge.Hub.Service.Startup.BuildContainer(IServiceCollection services) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Startup.cs:line 85
   at Microsoft.Azure.Devices.Edge.Hub.Service.Startup.ConfigureServices(IServiceCollection services) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Startup.cs:line 39
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Microsoft.Azure.Devices.Edge.Hub.Service.Hosting.Initialize(IConfigurationRoot configuration, X509Certificate2 serverCertificate, IDependencyManager dependencyManager, Boolean clientCertAuthEnabled) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Hosting.cs:line 34
   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 53
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 30
2020-09-15 14:48:09  Starting Edge Hub
2020-09-15 14:48:09.764 +00:00 Edge Hub Main()
<7> 2020-09-15 14:48:10.014 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to CreateServerCertificateAsync
<7> 2020-09-15 14:48:10.243 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2020-09-15 14:48:10.247 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2020-09-15 14:48:10.252 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/modules/%24edgeHub/genid/637352543162943819/certificate/server?api-version=2019-01-30
<7> 2020-09-15 14:48:10.362 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received Created
<7> 2020-09-15 14:48:10.444 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for CreateServerCertificateAsync
<7> 2020-09-15 14:48:10.717 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to TrustBundleAsync
<7> 2020-09-15 14:48:10.726 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2020-09-15 14:48:10.726 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2020-09-15 14:48:10.726 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/trust-bundle?api-version=2019-01-30
<7> 2020-09-15 14:48:10.738 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2020-09-15 14:48:10.741 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for TrustBundleAsync
<6> 2020-09-15 14:48:10.750 +00:00 [INF] [EdgeHub] - Installing certificates [CN=iotedged workload ca:12/02/2020 11:35:15],[CN=Test Edge Device CA:12/02/2020 11:35:15],[CN=Test Edge Owner CA:12/02/2020 11:35:15] to Root
<6> 2020-09-15 14:48:10.764 +00:00 [INF] [EdgeHub] - Installing certificates [CN=Test Edge Owner CA:12/02/2020 11:35:15] to Root

I don't really know all that much about linux/ubuntu, but apparently edgeHub does not have "out of the box write access", which would explain why the other modules went through. I tried to give group level access to the /tmp/ folder (chmod g+w /tmp/) but it did not help. The actual folder in the error (/tmp/edgeHub/edgeHub) does not exist and I assume this is because it's just a temporary installation folder or something like that, so I tried to make it myself and grant it the same group level access rights, but that did not help either.

I have also tried to uninstall edge and reinstall (thinking that all the previous problems during the installation made something go "boing"), but that lead to the same problem.

When looking at edgeAgent logs, it only shows that it's constantly trying to start edgeHub.

I have also run iotEdge check, but it shows no errors (8 warnings though).

*** edit *** I think I've narrowed it down to docker not having access to write to the edge storage folder, but so far all my tries in granting docker that access have failed. I can find tons of articles that tell me how to do it if I'm doing my own docker image, but thus far not one that tells me how to deal with an existing file access.

Answer 1

Turns out that MS installation document is not really comprehensive. The problem was, as suspected, the access rights of the edgeHub user. The installation instructions don't mention this at all, instead I found it in a IoT edge lab: https://microsoftlearning.github.io/AZ-220-Microsoft-Azure-IoT-Developer/Instructions/Labs/LAB_AK_14-iot-edge-device-in-restricted-network.html

However the commands in that lab don't work (or at least they did not work for me)

To make the story short, after IoT edge installation and configuration, you have to run the commands

sudo chown -R 1000:1000 [your local storageFolder]
sudo chmod -R 700 [your local storageFolder]

The UID 1000 = edgeHub user and the storageFolder can be found in your azure IoTHub -> edge module manifest. This will grant edgeHub docker the required access to the configured storage folder where it will save any offline data.