How to read a .properties file which contains keys that have a period character using Shell script

Question

I am trying to read a properties file from a shell script which contains a period (.) character like below:

# app.properties
db.uat.user=saple user
db.uat.passwd=secret

#/bin/sh
function pause(){
   read -p "$*"
}

file="./app.properties"

if [ -f "$file" ]
then
    echo "$file found."
 . $file

echo "User Id " $db.uat.user
echo "user password =" $db.uat.passwd
else
    echo "$file not found."
fi

I have tried to parse the file after sourcing the file but it is not working since the keys contains the "." character and there are spaces in that value also.

My properties file always resides in the same directory of the script or somewhere in /usr/share/doc

Answer 1

For a very high performance, and BASH 3.0 compatible solution:

file: loadProps.sh

function loadProperties() {
  local fileName=$1
  local prefixKey=$2

  if [ ! -f "${fileName}" ]; then
    echo "${fileName} not found!"
    return 1
  fi

  while IFS='=' read -r origKey value; do
    local key=${origKey}
    key=${key//[!a-zA-Z0-9_]/_} 
    if [[ "${origKey}" == "#"*   ]]; then
      local ignoreComments
    elif [ -z "${key}" ]; then
      local emptyLine
    else
      if [[ "${prefixKey}${key}" =~ ^[0-9].* ]]; then
        key=_${key}
      fi
      eval ${prefixKey}${key}=\${value}
    fi
  done < "${fileName}"
}

The other solutions provided here are great and elegant, but

• @fork2execve: slow when dealing with large properties files
• @Nicolai: slow when reading lots of properties
• @anubhava: require BASH 4.0 (for the array)

I needed something working on bash 3, dealing with properties files of ~1k entries, reading ~200 properties, and whole script called MANY times.

this function also deals with

• empty lines
• commented code
• duplicated entries (last one wins)
• normalize property names
• last line without a proper new line

Testing

file: my.properties

a=value
a=override value
b=what about `!@#$%^&*()_+[]\?
c=${a} no expansion
d=another = (equal sign)
e=     5 spaces front and back     
f=
#g=commented out
#ignore new line below

.@a%^=who named this???
a1=A-ONE
1a=ONE-A
X=lastLine with no new line!

test script

. loadProps.sh

loadProperties my.properties PROP_
echo "a='${PROP_a}'"
echo "b='${PROP_b}'"
echo "c='${PROP_c}'"
echo "d='${PROP_d}'"
echo "e='${PROP_e}'"
echo "f='${PROP_f}'"
echo "g='${PROP_g}'"
echo ".@a%^='${PROP___a__}'"
echo "a1='${PROP_a1}'"
echo "1a='${PROP_1a}'"
echo "X='${PROP_X}'"

loadProperties my.properties
echo "a='${a}'"
echo "1a='${_1a}'"

output

a='override value'
b='what about `!@#$%^&*()_+[]\?'
c='${a} no expansion'
d='another = (equal sign)'
e='     5 spaces front and back     '
f=''
g=''
.@a%^='who named this???'
a1='A-ONE'
1a='ONE-A'
X='lastLine with no new line!'
a='override value'
1a='ONE-A'

Performance Test

. loadProps.sh

function fork2execve() {
  while IFS='=' read -r key value; do
    key=$(echo $key | tr .-/ _ | tr -cd 'A-Za-z0-9_')
    eval ${key}=\${value}
  done < "$1"
}

function prop {
  grep '^\s*'"$2"'=' "$1" | cut -d'=' -f2-
}

function Nicolai() {
  for i in $(seq 1 $2); do 
    prop0000=$(prop $1 "property_0000")
  done
}

function perfCase() {
  echo "perfCase $1, $2, $3"
  time for i in $(seq 1 1); do 
    eval $1 $2 $3
  done
}

function perf() {
  perfCase $1 0001.properties $2
  perfCase $1 0010.properties $2
  perfCase $1 0100.properties $2
  perfCase $1 1000.properties $2
}

perf "loadProperties"
perf "fork2execve"
perf "Nicolai" 1
perf "Nicolai" 10
perf "Nicolai" 100

with 4 NNNN.properties files with entries such as

property_0000=value_0000
property_0001=value_0001
...
property_NNNN=value_NNNN

resulted with

function   , file,   #,     real,    user,     sys
loadPropert, 0001,    ,    0.058,   0.002,   0.005
loadPropert, 0010,    ,    0.032,   0.003,   0.005
loadPropert, 0100,    ,    0.041,   0.013,   0.006
loadPropert, 1000,    ,    0.140,   0.106,   0.013

fork2execve, 0001,    ,    0.053,   0.003,   0.007
fork2execve, 0010,    ,    0.211,   0.021,   0.051
fork2execve, 0100,    ,    2.146,   0.214,   0.531
fork2execve, 1000,    ,   21.375,   2.151,   5.312

Nicolai    , 0001,   1,    0.048,   0.003,   0.009
Nicolai    , 0010,   1,    0.047,   0.003,   0.009
Nicolai    , 0100,   1,    0.044,   0.003,   0.010
Nicolai    , 1000,   1,    0.044,   0.004,   0.009

Nicolai    , 0001,  10,    0.240,   0.020,   0.056
Nicolai    , 0010,  10,    0.263,   0.021,   0.059
Nicolai    , 0100,  10,    0.272,   0.023,   0.062
Nicolai    , 1000,  10,    0.295,   0.027,   0.059

Nicolai    , 0001, 100,    2.218,   0.189,   0.528
Nicolai    , 0010, 100,    2.213,   0.193,   0.537
Nicolai    , 0100, 100,    2.247,   0.196,   0.543
Nicolai    , 1000, 100,    2.323,   0.253,   0.534

Answer 2

Since variable names in the BASH shell cannot contain a dot or space it is better to use an associative array in BASH like this:

#!/bin/bash

# declare an associative array
declare -A arr

# read file line by line and populate the array. Field separator is "="
while IFS='=' read k v; do
   [[ -n $k ]] && arr["$k"]="$v"
done < app.properties

Testing:

Use declare -p to show the result:

declare -p arr  
      
declare -A arr='([db.uat.passwd]="secret" [db.uat.user]="saple user" )'

Answer 3

The mentioned solutions have many serious bugs and non of them worked for me.

• my value contains special characters like = or == at the end of the value
• my property file has comments

For example:

#encryption_key=dffsfsdfdsdfd
encryption_key=bdHY1uI1elN4RjZzwokJRw==

The results that the provided solutions returned are empty string or bdHY1uI1elN4RjZzwokJRw.

But my value is bdHY1uI1elN4RjZzwokJRw==.

The following solution reads value properly that contains any characters:

cat $properties_file | grep ^\\\s*${key}= | cut -d "=" -f 2-)

Bash method that you can use in your script:

# ------------------------------------------------------------------------------
# get a value from a properties file
#
# arguments:
#    arg-1: properties file
#    arg-2: key
# ------------------------------------------------------------------------------
function getValue {
   local properties_file key value
   properties_file="$1"
   key="$2"
   value=$(cat $properties_file | grep ^\\\s*${key}= | cut -d "=" -f 2-)
   printf "%s" "$value"
}

Answer 4

I use simple grep inside function in bash script to receive properties from .properties file.

This properties file I use in two places - to setup dev environment and as application parameters.

I believe that grep may work slow in big loops but it solves my needs when I want to prepare dev environment.

Hope, someone will find this useful.

Example:

File: setup.sh

#!/bin/bash

ENV=${1:-dev}

# Reads the value of a property from a properties file.
#
# $1 - Key name, matched at beginning of line.
function prop {
    grep "^${1}" env/${ENV}.properties|cut -d'=' -f2
}

docker create \
    --name=myapp-storage \
    -p $(prop 'app.storage.address'):$(prop 'app.storage.port'):9000 \
    -h $(prop 'app.storage.host') \
    -e STORAGE_ACCESS_KEY="$(prop 'app.storage.access-key')" \
    -e STORAGE_SECRET_KEY="$(prop 'app.storage.secret-key')" \
    -e STORAGE_BUCKET="$(prop 'app.storage.bucket')" \
    -v "$(prop 'app.data-path')/storage":/app/storage \
    myapp-storage:latest

docker create \
    --name=myapp-database \
    -p "$(prop 'app.database.address')":"$(prop 'app.database.port')":5432 \
    -h "$(prop 'app.database.host')" \
    -e POSTGRES_USER="$(prop 'app.database.user')" \
    -e POSTGRES_PASSWORD="$(prop 'app.database.pass')" \
    -e POSTGRES_DB="$(prop 'app.database.main')" \
    -e PGDATA="/app/database" \
    -v "$(prop 'app.data-path')/database":/app/database \
    postgres:9.5

File: env/dev.properties

app.data-path=/apps/myapp/

#==========================================================
# Server properties
#==========================================================
app.server.address=127.0.0.70
app.server.host=dev.myapp.com
app.server.port=8080

#==========================================================
# Backend properties
#==========================================================
app.backend.address=127.0.0.70
app.backend.host=dev.myapp.com
app.backend.port=8081
app.backend.maximum.threads=5

#==========================================================
# Database properties
#==========================================================
app.database.address=127.0.0.70
app.database.host=database.myapp.com
app.database.port=5432
app.database.user=dev-user-name
app.database.pass=dev-password
app.database.main=dev-database

#==========================================================
# Storage properties
#==========================================================
app.storage.address=127.0.0.70
app.storage.host=storage.myapp.com
app.storage.port=4569
app.storage.endpoint=http://storage.myapp.com:4569
app.storage.access-key=dev-access-key
app.storage.secret-key=dev-secret-key
app.storage.region=us-east-1
app.storage.bucket=dev-bucket

Usage:

./setup.sh dev

Answer 5

I think Nicolai's answer is good. However, sometimes people write

app.server.address = 127.0.0.70

instead of

app.server.address=127.0.0.70

In this situation. If we directly use

function prop {
    grep "${1}" env/${ENV}.properties|cut -d'=' -f2
}

it will produce " 127.0.0.70" instead of "127.0.0.70", take some error in string combination. To solve this, we could add "| xargs". And it will be

grep "${1}" ${ENV}.properties|cut -d'=' -f2 | xargs

And we will get what we want.

Answer 6

I found using while IFS='=' read -r to be a bit slow (I don't know why, maybe someone could briefly explain in a comment or point to a SO answer?). I also found @Nicolai answer very neat as a one-liner, but very inefficient as it will scan the entire properties file over and over again for every single call of prop.

I found a solution that answers the question, performs well and it is a one-liner (bit verbose line though).

The solution does sourcing but massages the contents before sourcing:

#!/usr/bin/env bash

source <(grep -v '^ *#' ./app.properties | grep '[^ ] *=' | awk '{split($0,a,"="); print gensub(/\./, "_", "g", a[1]) "=" a[2]}')

echo $db_uat_user

Explanation:

grep -v '^ *#': discard comment lines grep '[^ ] *=': discards lines without = split($0,a,"="): splits line at = and stores into array a, i.e. a[1] is the key, a[2] is the value gensub(/\./, "_", "g", a[1]): replaces . with _ print gensub... "=" a[2]} concatenates the result of gensub above with = and value.

Edit: As others pointed out, there are some incompatibilities issues (awk) and also it does not validate the contents to see if every line of the property file is actually a kv pair. But the goal here is to show the general idea for a solution that is both fast and clean. Sourcing seems to be the way to go as it loads the properties once that can be used multiple times.

Answer 7

As (Bourne) shell variables cannot contain dots you can replace them by underscores. Read every line, translate . in the key to _ and evaluate.

#/bin/sh

file="./app.properties"

if [ -f "$file" ]
then
  echo "$file found."

  while IFS='=' read -r key value
  do
    key=$(echo $key | tr '.' '_')
    eval ${key}=\${value}
  done < "$file"

  echo "User Id       = " ${db_uat_user}
  echo "user password = " ${db_uat_passwd}
else
  echo "$file not found."
fi

Note that the above only translates . to _, if you have a more complex format you may want to use additional translations. I recently had to parse a full Ant properties file with lots of nasty characters, and there I had to use:

key=$(echo $key | tr .-/ _ | tr -cd 'A-Za-z0-9_')

Answer 8

@fork2x

I have tried like this .Please review and update me whether it is right approach or not.

#/bin/sh
function pause(){
   read -p "$*"
}

file="./apptest.properties"


if [ -f "$file" ]
then
    echo "$file found."

dbUser=`sed '/^\#/d' $file | grep 'db.uat.user'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`
dbPass=`sed '/^\#/d' $file | grep 'db.uat.passwd'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`

echo database user = $dbUser
echo database pass = $dbPass

else
    echo "$file not found."
fi