CODEWITHSUNDEEP
pythonconfigurationenvironment-variables.envdynaconf
Laodao
Laodao

Reputation: 1709

Replacing the placeholder in a hierarchical config.yaml file with the value in the .env file

Right now, I have config.yaml like this

kafka:
  bootstrap_servers: "${BOOTSTRAP_SERVERS}"
  group: "${GROUP_NAME_1}"
  topics: 
    - name: "${TOPIC_1}"
      consumers: "${CONSUMER_NUMBER_FOR_TOPIC_1}"

And I have Dynaconf working as below:

from dynaconf import Dynaconf
from dynaconf.validator import Validator

settings = Dynaconf(
    envvar_prefix="service-a",
    settings_files=['config.yaml'],  
    load_dotenv=True,
    dotenv_path='.env',
    validators=[
        Validator(
            "server.port", must_exist=True
        )
    ]
)

Then, the settings will be used like

def start_kafka_consumers():
    topics = settings.kafka.topics
    threads = []
    for topic in topics:
        topic_name = topic['name']
        consumer_count = topic['consumers']
        logger.info(f"{consumer_count} consumers will be started for topic {topic_name}")

        for _ in range(consumer_count):
            thread = threading.Thread(target=start_consumer_for_topic, args=(topic_name,))
            thread.start()
            threads.append(thread)

The beauty of using the config.yaml file is that I can group some properties with hierarchy. For example, I can have topic "abc" with 1 consumer grouped under topics. And later, I can simply add topic "def" with 2 consumers. And the code will dynamically load and use them.

The reason I want to use the place holder and an .env file to define the acutal value is because, I want to run it on my local with the corresponding value. Then, later, when it is uploaded to GCP cloud run, I can use environment variables (e.g. secret manger) to overwrite the value.

However, the ${} placeholder doesn't work as expected. The Dynaconf doesn't use the .env file value to replace the placeholder text. Any suggestions? Thanks.

Upvotes: 0

Views: 35

Answers (1)

Laodao
Laodao

Reputation: 1709

I find the solution, I can use the Dynaconf to load the config.yaml file, which will load the env variable name. And then read that variable using this name from .env file. I am not sure if there is any other approaches.

In normal case, it's not worthy. But as I mentioned in my question, for my case, the hierarchy of the config.yaml file provide me a flexible approach to instantiate the consumer for multiple topics.

Upvotes: 0

Related Questions