How to setup celery, celerybeat, flower and Fastapi in devcontainer such that celery containers should only start on some actions

Question

This can be a long context, so please bear with me. I have a fastapi app that schedules tasks to be performed in background. The background tasks are supposed to be run every 5 minutes. The catch is that, most of the time I am developing, I don't want the background tasks to run (aka celery to be active). I need a way to start and stop it when I wish. I am using Visual Studio Code. Here's what I have tried so far

naive devcontainers

My .devcontainer/docker-compose.yaml looks like this

services:
  trader:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
      - ~/.ssh:/home/monkey/.ssh:ro  # Mount SSH keys
      - ~/.gitconfig:/home/monkey/.gitconfig:cached
      - ruff_cache:/app/.ruff_cache
    command: sleep infinity
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    depends_on:
      - redis
      - postgres

  celery:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
    command: celery -A src.celery worker --loglevel=debug
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    depends_on:
      - redis
      - postgres

  celerybeat:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
    command: celery -A src.celery beat --loglevel=debug
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    depends_on:
      - redis
      - postgres

  redis:
    image: redis:latest
    networks:
      - trader-network

  postgres:
    build:
      context: .
      dockerfile: Dockerfile.postgres
      args:
        REBUILD_DATE: ${REBUILD_DATE:-1}
    environment:
      - POSTGRES_USER=myuser
      - POSTGRES_PASSWORD=mypassword
      - POSTGRES_MULTIPLE_DATABASES=db
    volumes:
      - postgres_data_trader:/var/lib/postgresql/data
      - ./scripts/init_multiple_db.sh:/docker-entrypoint-initdb.d/init_multiple_db.sh
      - ./scripts/pg_custom_entrypoint.sh:/usr/local/bin/custom-entrypoint.sh
    ports:
      - "6543:5432" # Expose the port to the host machine for debugging
    networks:
      - trader-network
    entrypoint: ["/usr/local/bin/custom-entrypoint.sh"]
    command: ["postgres"]

  flower:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
    ports:
      - "5555:5555"  # Expose Flower's web interface
    command: celery -A src.celery flower --port=5555 --broker=redis://redis:6379/0
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    depends_on:
      - redis
      - postgres

networks:
  trader-network:
    driver: bridge

volumes:
  ruff_cache:
    driver: local
  postgres_data_trader:
    driver: local

And my devcontainer.json looks like this

{
  "name": "Python 3.12.3",
  "dockerComposeFile": ["./docker-compose.yml"],
  "service": "trader",
  "workspaceFolder": "/app/",
  "customizations": {
      "vscode": {
          "settings": {
              "python.defaultInterpreterPath": "/usr/local/bin/python",
              "python.linting.enabled": true,
              "python.linting.pylintEnabled": true,
              "python.formatting.autopep8Path": "/usr/local/bin/autopep8",
              "python.formatting.blackPath": "/usr/local/bin/black",
              "python.formatting.yapfPath": "/usr/local/bin/yapf",
              "python.linting.banditPath": "/usr/local/bin/bandit",
              "python.linting.flake8Path": "/usr/local/bin/flake8",
              "python.linting.mypyPath": "/usr/local/bin/mypy",
              "python.linting.pycodestylePath": "/usr/local/bin/pycodestyle",
              "python.linting.pydocstylePath": "/usr/local/bin/pydocstyle"
          },
          "extensions": [
            "ms-python.python",
            "ms-python.vscode-pylance",
            "ms-toolsai.jupyter",
            "ms-azuretools.vscode-docker",
            "github.vscode-github-actions",
            "vscodevim.vim"
          ]
      }
  },
  "forwardPorts": [8000, 5432, 5555],
  "remoteUser": "monkey",
  "overrideCommand": false,
  "runServices": ["trader", "redis", "postgres"]
}

This would start ALL the services i.e. trader, postgres, redis, celery, celerybeat, flower etc at once when I was inside devcontainer. The issue is that I am using a few paid APIs that i dont need to always do with my celery while developing.

devcontainer with profiles

In this method, I created a profile such that by default only trader, redis and postgres would turn on. Then I setup docker socket so that when i run a command from within my trader devcontainer, it would spin containers in host machine for celery and flower etc.

services:
  trader:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
      - ~/.ssh:/home/monkey/.ssh:ro  # Mount SSH keys
      - ~/.gitconfig:/home/monkey/.gitconfig:cached
      - ruff_cache:/app/.ruff_cache
    command: sleep infinity
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    depends_on:
      - redis
      - postgres
    profiles: ["default"]

  celery:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
    command: celery -A src.celery worker --loglevel=debug
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    profiles: ["optional"]

  celerybeat:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
    command: celery -A src.celery beat --loglevel=debug
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    profiles: ["optional"]

  redis:
    image: redis:latest
    networks:
      - trader-network
    profiles: ["default"]

  postgres:
    build:
      context: .
      dockerfile: Dockerfile.postgres
      args:
        REBUILD_DATE: ${REBUILD_DATE:-1}
    environment:
      - POSTGRES_USER=myuser
      - POSTGRES_PASSWORD=mypassword
      - POSTGRES_MULTIPLE_DATABASES=db
    volumes:
      - postgres_data_trader:/var/lib/postgresql/data
      - ./scripts/init_multiple_db.sh:/docker-entrypoint-initdb.d/init_multiple_db.sh
      - ./scripts/pg_custom_entrypoint.sh:/usr/local/bin/custom-entrypoint.sh
    ports:
      - "6543:5432" # Expose the port to the host machine for debugging
    networks:
      - trader-network
    entrypoint: ["/usr/local/bin/custom-entrypoint.sh"]
    command: ["postgres"]
    profiles: ["default"]

  flower:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/app:cached
    ports:
      - "5555:5555"  # Expose Flower's web interface
    command: celery -A src.celery flower --port=5555 --broker=redis://redis:6379/0
    environment:
      - AGENT_CACHE_REDIS_HOST=redis
      - DB_URL=postgresql://myuser:mypassword@postgres:5432/db
    env_file:
      - .env.local
    networks:
      - trader-network
    profiles: ["optional"]

networks:
  trader-network:
    driver: bridge

volumes:
  ruff_cache:
    driver: local
  postgres_data_trader:
    driver: local

and my devcontainer.json being

{
  "name": "Python 3.12.3",
  "dockerComposeFile": ["./docker-compose.yml"],
  "service": "trader",
  "workspaceFolder": "/app/",
  "customizations": {
      "vscode": {
          "settings": {
              "python.defaultInterpreterPath": "/usr/local/bin/python",
              "python.linting.enabled": true,
              "python.linting.pylintEnabled": true,
              "python.formatting.autopep8Path": "/usr/local/bin/autopep8",
              "python.formatting.blackPath": "/usr/local/bin/black",
              "python.formatting.yapfPath": "/usr/local/bin/yapf",
              "python.linting.banditPath": "/usr/local/bin/bandit",
              "python.linting.flake8Path": "/usr/local/bin/flake8",
              "python.linting.mypyPath": "/usr/local/bin/mypy",
              "python.linting.pycodestylePath": "/usr/local/bin/pycodestyle",
              "python.linting.pydocstylePath": "/usr/local/bin/pydocstyle"
          },
          "extensions": [
            "ms-python.python",
            "ms-python.vscode-pylance",
            "ms-toolsai.jupyter",
            "ms-azuretools.vscode-docker",
            "github.vscode-github-actions",
            "vscodevim.vim"
          ]
      }
  },
  "mounts": [
    "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind"
  ],
  "forwardPorts": [8000, 5432, 5555],
  "remoteUser": "monkey",
  "overrideCommand": false,
  "runServices": ["trader", "redis", "postgres"]
}

The issue is that for some reason, when I start celery containers, I keep getting the error

[+] Running 3/0
 ✔ Container devcontainer-celerybeat-1  Created                                                                                           0.0s 
 ✔ Container devcontainer-flower-1      Created                                                                                           0.0s 
 ✔ Container devcontainer-celery-1      Created                                                                                           0.0s 
Attaching to celery-1, celerybeat-1, flower-1
celery-1      | Usage: celery [OPTIONS] COMMAND [ARGS]...
celery-1      | Try 'celery --help' for help.
celery-1      | 
celery-1      | Error: Invalid value for '-A' / '--app': 
celery-1      | Unable to load celery application.
celery-1      | The module src.celery was not found.
flower-1      | Usage: celery [OPTIONS] COMMAND [ARGS]...
flower-1      | Try 'celery --help' for help.
flower-1      | 
flower-1      | Error: Invalid value for '-A' / '--app': 
flower-1      | Unable to load celery application.
flower-1      | The module src.celery was not found.
celerybeat-1  | Usage: celery [OPTIONS] COMMAND [ARGS]...
celerybeat-1  | Try 'celery --help' for help.
celerybeat-1  | 
celerybeat-1  | Error: Invalid value for '-A' / '--app': 
celerybeat-1  | Unable to load celery application.
celerybeat-1  | The module src.celery was not found.
celery-1 exited with code 2
celerybeat-1 exited with code 2
flower-1 exited with code 2

So I am quite confused (and also frustrated). Of course I have tried all GPTs and Claudes but I think this is one of those problem that needs a human brain to figure out. So please help me.