CODEWITHSUNDEEP
pythonpandas
Daniel Hutchinson
Daniel Hutchinson

Reputation: 165

Python & Pandas: appending data to new column

With Python and Pandas, I'm writing a script that passes text data from a csv through the pylanguagetool library to calculate the number of grammatical errors in a text. The script successfully runs, but appends the data to the end of the csv instead of to a new column.

The structure of the csv is:

CSV1

The working code is:

import pandas as pd
from pylanguagetool import api

df = pd.read_csv("Streamlit\stack.csv")

text_data = df["text"].fillna('')
length1 = len(text_data)

for i, x in enumerate(range(length1)):
    # this is the pylanguagetool operation
    errors = api.check(text_data, api_url='https://languagetool.org/api/v2/', lang='en-US')
    result = str(errors)
    # this pulls the error count "message" from the pylanguagetool json
    error_count = result.count("message")
    output_df = pd.DataFrame({"error_count": [error_count]})
    output_df.to_csv("Streamlit\stack.csv", mode="a", header=(i == 0), index=False)

The output is:

CSV2

Expected output:

CSV3

What changes are necessary to append the output like this?

Upvotes: 0

Views: 100

Answers (2)

not_speshal
not_speshal

Reputation: 23146

Instead of using a loop, you might consider lambda which would accomplish what you want in one line:

df["error_count"] = df["text"].fillna("").apply(lambda x: len(api.check(x, api_url='https://languagetool.org/api/v2/', lang='en-US')["matches"]))

>>> df
   user_id  ... error_count
0       10  ...           2
1       11  ...           0
2       12  ...           0
3       13  ...           0
4       14  ...           0
5       15  ...           2

Edit:

You can write the above to a .csv file with:

df.to_csv("Streamlit\stack.csv", index=False)

You don't want to use mode="a" as that opens the file in append mode whereas you want (the default) write mode.

Upvotes: 4

Muhammad Rasel
Muhammad Rasel

Reputation: 724

My strategy would be to keep the error counts in a list then create a separate column in the original database and finally write that database to csv:

text_data = df["text"].fillna('')
length1 = len(text_data)
error_count_lst = []
for i, x in enumerate(range(length1)):
    errors = api.check(text_data, api_url='https://languagetool.org/api/v2/', lang='en-US')
    result = str(errors)
    error_count = result.count("message")
    error_count_lst.append(error_count)

text_data['error_count'] = error_count_lst
text_data.to_csv('file.csv', index=False)

Upvotes: 1

Related Questions