CODEWITHSUNDEEP
pythonlinuxazure
bermz kastral
bermz kastral

Reputation: 93

install python package in azure batch node

How can i manage to install python package in azure batch node like pip install azure-storage-blob. I tried to write that command in start task but i causes the stark task to fail?

Upvotes: 1

Views: 3244

Answers (1)

Nic3500
Nic3500

Reputation: 8611

When you define your pool, you must specify a pool start task. This start task will be run on each node as it is created and joins the pool (or when it is rebooted, re-imaged).

You can do it in the console under your pool. Or if you used Python, do it like this (this is an example, adjust as required!). Look at how the task_commands variable is defined (the curl is requried):

def create_pool(batch_service_client, pool_id,
            resource_files, publisher, offer, sku):
"""
Creates a pool of compute nodes with the specified OS settings.

:param batch_service_client: A Batch service client.
:type batch_service_client: `azure.batch.BatchServiceClient`
:param str pool_id: An ID for the new pool.
:param list resource_files: A collection of resource files for the pool's
start task.
:param str publisher: Marketplace image publisher
:param str offer: Marketplace image offer
:param str sku: Marketplace image sku
"""
print('Creating pool [{}]...'.format(pool_id))

# Create a new pool of Linux compute nodes using an Azure Virtual Machines
# Marketplace image. For more information about creating pools of Linux
# nodes, see:
# https://azure.microsoft.com/documentation/articles/batch-linux-nodes/

# Specify the commands for the pool's start task. The start task is run
# on each node as it joins the pool, and when it's rebooted or re-imaged.
# We use the start task to prep the node for running our task script.
task_commands = [
    'cp -p {} $AZ_BATCH_NODE_SHARED_DIR'.format('YOUR-PYTHON-SCRIPT.py'),

    # Install pip
    'curl -fSsL https://bootstrap.pypa.io/get-pip.py | python',
    # Install the azure-storage module so that the task script can access
    # Azure Blob storage, pre-cryptography version
    'pip install azure-storage==0.32.0']

# Get the node agent SKU and image reference for the virtual machine
# configuration.
# For more information about the virtual machine configuration, see:
# https://azure.microsoft.com/documentation/articles/batch-linux-nodes/
sku_to_use, image_ref_to_use = \
    helpers.select_latest_verified_vm_image_with_node_agent_sku(
        batch_service_client, publisher, offer, sku)

user = batchmodels.AutoUserSpecification(
    scope=batchmodels.AutoUserScope.pool,
    elevation_level=batchmodels.ElevationLevel.admin)

new_pool = batch.models.PoolAddParameter(
    id=pool_id,
    virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
        image_reference=image_ref_to_use,
        node_agent_sku_id=sku_to_use),
    vm_size=_POOL_VM_SIZE,
    target_dedicated_nodes=_POOL_NODE_COUNT,
    start_task=batch.models.StartTask(
        command_line=helpers.wrap_commands_in_shell('linux',
                                                           task_commands),
        user_identity=batchmodels.UserIdentity(auto_user=user),
        wait_for_success=True,
        resource_files=resource_files),
)

try:
    batch_service_client.pool.add(new_pool)
except batchmodels.batch_error.BatchErrorException as err:
    print_batch_exception(err)
    raise

Disclaimer: based on python_tutorial_client.py from Microsoft, look it up, it will get you started!

Upvotes: 4

Related Questions