CODEWITHSUNDEEP
dockerapache-sparkairflow
DBA108642
DBA108642

Reputation: 2112

Change container port in docker compose

I am trying to get a docker-compose file working with both airflow and spark. Airflow typically runs on 8080:8080 which is needed by spark as well. I have the following docker-compose file:

version: '3.7'
services:
    master:
      image: gettyimages/spark
      command: bin/spark-class org.apache.spark.deploy.master.Master -h master
      hostname: master
      environment:
        MASTER: spark://master:7077
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: localhost
      expose:
        - 7001
        - 7002
        - 7003
        - 7004
        - 7005
        - 7077
        - 6066
      ports:
        - 4040:4040
        - 6066:6066
        - 7077:7077
        - 8080:8080
      volumes:
        - ./conf/master:/conf
        - ./data:/tmp/data

    worker:
      image: gettyimages/spark
      command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
      hostname: worker
      environment:
        SPARK_CONF_DIR: /conf
        SPARK_WORKER_CORES: 2
        SPARK_WORKER_MEMORY: 1g
        SPARK_WORKER_PORT: 8881
        SPARK_WORKER_WEBUI_PORT: 8081
        SPARK_PUBLIC_DNS: localhost
      links:
        - master
      expose:
        - 7012
        - 7013
        - 7014
        - 7015
        - 8881
      ports:
        - 8081:8081
      volumes:
        - ./conf/worker:/conf
        - ./data:/tmp/data
    postgres:
        image: postgres:9.6
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow
        logging:
            options:
                max-size: 10m
                max-file: "3"

    webserver:
        image: puckel/docker-airflow:1.10.9
        restart: always
        depends_on:
            - postgres
        environment:
            - LOAD_EX=y
            - EXECUTOR=Local
        logging:
            options:
                max-size: 10m
                max-file: "3"
        volumes:
            - ./dags:/usr/local/airflow/dags
            # Add this to have third party packages
            - ./requirements.txt:/requirements.txt
            # - ./plugins:/usr/local/airflow/plugins
        ports:
            - "8082:8080" # NEED TO CHANGE THIS LINE
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3

but specifically need to change the line:

ports:
    - "8082:8080" # NEED TO CHANGE THIS LINE

under web server so there's no port conflict. However when I change the container port to something other than 8080:8080 it doesnt work (cannot connect/find server). How do I change the container port successfully?

Upvotes: 1

Views: 4039

Answers (1)

ItayB
ItayB

Reputation: 11337

When you specified port mapping in docker you are giving 2 ports, for instance: 8082:8080. The right port is the one that is listening within the container. You can have several containers that listening internally to the same port. They are still not available in your localhost - that's why we use the ports section for.

Now, in your localhost you cannot bind the same port more than once. That's why docker failed when you try to set on the left side 8080 more than one time. In your current compose file, the spark service port is mapped to 8080 (left side of 8080:8080) and the webserver service is mapped to 8082 (left side of 8082:8080).

if you want to access spark, goto: http://localhost:8080 and for the web server, go to http://localhost:8082

Upvotes: 3

Related Questions