sam build --use-container failed but sam build is success

Question

The issue happens in the following setup:

1. Virtual Machine launched in OpenStack
2. OS is Ubuntu 16.04 LTS
3. Python version 3.7.6 with virtualenv installed
4. SAM CLI version 0.39.0

To replicate the issues, you may use the above setup and perform the following steps:

1. $ sam init
2. $ Choice: 1
3. $ Runtime 9 Select python3.7
4. $ Project name[sam-app]: sam-app
5. $ Template selection: 1 Select Hello World Example

Wait for the application to be generated.

6. $ cd sam-app
7. $ python3 -m virtualenv venv
8. $ source venv/bin/activate
9. $(venv) source venv/bin/activate
10. $(venv) sam build

The following output shall appear:

Building resource 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

However, if the --use-container flag is used, the following error will appear

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Starting Build inside a container
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
Building resource 'HelloWorldFunction'

Fetching lambci/lambda:build-python3.7 Docker container image......
Mounting /home/ubuntu/test/sam-app/hello_world as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    requirements_path=self.manifest_path,
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 198, in build_site_packages
    wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 222, in _download_dependencies
    deps = self._download_all_dependencies(requirements_filename, directory)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 305, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 594, in download_all_dependencies
    raise NoSuchPackageError(str(package_name))
aws_lambda_builders.workflows.python_pip.packager.NoSuchPackageError: Could not satisfy the requirement: requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Could not satisfy the requirement: requests
Builder workflow failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    requirements_path=self.manifest_path,
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 198, in build_site_packages
    wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 222, in _download_dependencies
    deps = self._download_all_dependencies(requirements_filename, directory)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 305, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 594, in download_all_dependencies
    raise NoSuchPackageError(str(package_name))
aws_lambda_builders.workflows.python_pip.packager.NoSuchPackageError: Could not satisfy the requirement: requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Could not satisfy the requirement: requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/__main__.py", line 126, in main
    mode=params.get("mode", None),
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/builder.py", line 125, in build
    return workflow.run()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 76, in wrapper
    func(self, *args, **kwargs)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 276, in run
    raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests
Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests"}}

Build Failed
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 107282, 'exitReason': 'BuildError', 'exitCode': 1, 'requestId': '62d1fc73-70e5-4592-8c78-8fa273684592', 'installationId': 'ce8ffa14-684f-4628-97fe-288848fcf73d', 'sessionId': '4ca5bb42-5cf1-4ae5-89ad-9ed00366fefb', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '0.39.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests

Answer 1

The above issues arise due to python PythonPipBuilder was not able to resolve the dependence. I had since log into docker and install the dependence manually, subsequently I suspected it was due to docker network issues.

Subsequently, the sam application was built by adding in additional flag which enable docker to use host network.

$(venv) sam build --use-container --docker-network host --debug

The output of the command:

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Starting Build inside a container
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
Building resource 'HelloWorldFunction'

Fetching lambci/lambda:build-python3.7 Docker container image......
Mounting /home/ubuntu/test/sam-app/hello_world as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch
Full dependency closure: {requests==2.23.0(wheel), certifi==2019.11.28(wheel), urllib3==1.25.8(wheel), chardet==3.0.4(wheel), idna==2.9(wheel)}
initial compatible: {requests==2.23.0(wheel), certifi==2019.11.28(wheel), urllib3==1.25.8(wheel), chardet==3.0.4(wheel), idna==2.9(wheel)}
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: {requests==2.23.0(wheel), certifi==2019.11.28(wheel), urllib3==1.25.8(wheel), chardet==3.0.4(wheel), idna==2.9(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {requests==2.23.0(wheel), certifi==2019.11.28(wheel), urllib3==1.25.8(wheel), chardet==3.0.4(wheel), idna==2.9(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {requests==2.23.0(wheel), certifi==2019.11.28(wheel), urllib3==1.25.8(wheel), chardet==3.0.4(wheel), idna==2.9(wheel)}
Final compatible: {chardet==3.0.4(wheel), requests==2.23.0(wheel), certifi==2019.11.28(wheel), urllib3==1.25.8(wheel), idna==2.9(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
Build inside container was successful. Copying artifacts from container to host
Copying from container: /tmp/samcli/artifacts/. -> /home/ubuntu/test/sam-app/.aws-sam/build/HelloWorldFunction
Build inside container succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 5544, 'exitReason': 'success', 'exitCode': 0, 'requestId': 'bd9e7b0a-82ac-40a8-a172-a2d49f0633ff', 'installationId': 'ce8ffa14-684f-4628-97fe-288848fcf73d', 'sessionId': '48762175-e559-4759-b2fa-6a66822e381e', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '0.39.0'}}]}

Credit also goes to my colleague Dr. Phetsouvanh Silivanxay to troubleshoot the issue together.