CODEWITHSUNDEEP
pythonpandascsvfile
Rahul Tyagi
Rahul Tyagi

Reputation: 25

Inconsistent behavior seen in Target file after performing append operations

I am facing a strange issue while appending the entries to a File.

A bit of background about code, it's a set of Python Modules whereby,

+----+-----------+---------------+----------------+------------+----------+--------------------------------------+--------------------------------------------+
|    | UE_NAME   | UE_TYPE       | ES_NAME        | APN_NAME   | STATUS   | UE_ROUTES_OPERATION(MOBILE_ROUTER)   | UE_ROUTES(MOBILE_ROUTER)                   |
+====+===========+===============+================+============+==========+======================================+============================================+
|  0 | ue_1      | MOBILE_ROUTER | Edge_service_4 | apn1862    | UE_ADDED | UE_ROUTE_ADDED                       | ['172.16.10.0/26(A)']                      |
+----+-----------+---------------+----------------+------------+----------+--------------------------------------+--------------------------------------------+
|  1 | ue_6      | IOT_MOBILE    | Edge_service_1 | apn_nuage  | UE_ADDED | --                                   | --                                         |
+----+-----------+---------------+----------------+------------+----------+--------------------------------------+--------------------------------------------+
|  2 | ue_2      | MOBILE_ROUTER | Edge_service_4 | apn1862    | UE_ADDED | UE_ROUTE_ADDED                       | ['172.16.10.0/18(A)', '172.16.12.0/24(A)'] |
+----+-----------+---------------+----------------+------------+----------+--------------------------------------+--------------------------------------------+
|  3 | ue_201    | MOBILE_ROUTER | test1          | apn_nuage  | UE_ADDED | UE_ROUTE_ADDED                       | ['172.16.10.0/16 (A)']                     |
+----+-----------+---------------+----------------+------------+----------+--------------------------------------+--------------------------------------------+

Now, coming ack to problem, everything is working fine from code perspective. But, I see an issue with updates getting appended to Master_record file. After initial execution of this Tool, Following Master record CSV was seen,

more .\master_records.csv

name,IMSI,MSISDN,ICCID,IMEI,Enterprise,operationalStatus,ueType,ueRoutes,description,apn_name
ue_1,1,1,1,1,Edge_service_4,IDLE,MOBILE_ROUTER,"172.16.10.0/26",UE-1-desc,apn1862
ue_6,6,6,6,6,Edge_service_1,CONNECTED,IOT_MOBILE,,UE-6,apn_nuage
ue_2,2,2,2,2,Edge_service_4,CONNECTED,MOBILE_ROUTER,"172.16.10.0/18,172.16.12.0/24",UE-2-desc,apn1862
ue_201,201,201,201,203,test1,IDLE,MOBILE_ROUTER,"172.16.10.0/16 ",UE-201,apn_nuage

But, After modifying one of the entries[say, ue_1] in "Latest_records.csv", double-quotes went missing from the Object records of ue_201, and after completion of script, master_records file stood at,

more .\master_records.csv

name,IMSI,MSISDN,ICCID,IMEI,Enterprise,operationalStatus,ueType,ueRoutes,description,apn_name
ue_6,6,6,6,6,Edge_service_1,CONNECTED,IOT_MOBILE,,UE-6,apn_nuage
ue_2,2,2,2,2,Edge_service_4,CONNECTED,MOBILE_ROUTER,"172.16.10.0/18,172.16.12.0/24",UE-2-desc,apn1862
ue_201,201,201,201,203,test1,IDLE,MOBILE_ROUTER,172.16.10.0/16 ,UE-201,apn_nuage
ue_1,1,1,1,1,Edge_service_4,CONNECTED,MOBILE_ROUTER,"172.16.10.0/26",UE-1-desc,apn1862

You can see that double quotes around the 172.16.10.0/16 are missing for object, ue_201

This issue is seen only in the instances where there is only one object under 'ueRoutes' column, If I have multiple Objects like "172.16.10.0/18,172.16.12.0/24", I don't see this issue.

Code to update the Master Records is very simple as marked below,

def add_entry_master_record(row_entry):
"""
Adds a UE Entry to Master Record file.

:param row_entry: UE record Entry.

:return: Updates the existing Master Record file.
"""

try:
    with open(m_record, 'a+') as master_record_update:
        master_record_update.write(row_entry)
        master_record_update.write('\n')

except FileNotFoundError as err_file:
    raise SystemExit(err_file)

Upon checking the contents of 'row_entry', I don't see any updates going to Master_record file for ue_201 and it was only for ue_1 something like below,

row_entry = ue_1,1,1,1,1,Edge_service_4,CONNECTED,MOBILE_ROUTER,"172.16.10.0/26",UE-1-desc,apn1862

Once this irregularity creeps in, it starts messing with my code as the delta calculations between latest_record and master_record file dishes out the not-delta content as well. I was able to work-around this issue by using some regex substitutions before comparison, but I am still curious about why this irregularity happens in first place.

A note on how I update the Master_record file:

    def drop_rows_master_record(ue_name, mod_entry):
    """
    Drops the Entries from Master Record csv file.

    :param ue_name: UE Name for which Entry needs to be dropped.
    :param mod_entry: UE Record

    :return: Drops the deleted/modified UE record from Master Records.
    """

    df = pd.read_csv(m_record)
    df_filtered = df[(df['name'].str.lower() == ue_name.lower()) &
                     (df['Enterprise'].str.lower() == mod_entry['Enterprise'].lower())]
    df.drop(df_filtered.index, inplace=True)
    df.reset_index(drop=True, inplace=True)
    df.to_csv(m_record, index=False)

Let me know, if you guys can share a pointer on why this discrepancy is creeping in, in first place.

Upvotes: 0

Views: 50

Answers (1)

Rahul Tyagi
Rahul Tyagi

Reputation: 25

I found the root of this issue. It wasn't an issue with Pandas or Python. :) Pandas was just confirming to the CSV format and removing the quotes if there was a single element defined inside a column.

So, for example, if a CSV column has only one Value like

ue_routes
10.18.19.0/24

in text file, we will see it without quotes.

But, if there are multiple values for a CSV column,

ue_routes
10.18.19.0/24, 10.18.20.0/24

in text file, we will see it with quotes.

Upvotes: 1

Related Questions