CODEWITHSUNDEEP
pythonpandas
d1r3w0lF
d1r3w0lF

Reputation: 85

Pandas append() error with two dataframes

When I try to append two or more dataframe and output the result to a csv, it shows like a waterfall format.

    dataset = pd.read_csv('testdata.csv')
  for i in segment_dist:
      for j in step:
          print_msg = str(i) + ":" + str(j)
          print("\n",i,":",j,"\n")
          temp = pd.DataFrame(estimateRsq(dataset,j,i),columns=[print_msg])
          csv = csv.append(temp)
  csv.to_csv('output.csv',encoding='utf-8', index=False)

estimateRsq() returns array. I think this much code snippet should be enough to help me out.

The format I am getting in output.csv is: enter image description here

Please help, How can I shift the contents go up from index 1.

Upvotes: 2

Views: 1269

Answers (2)

jpp
jpp

Reputation: 164713

Build a list of dataframes, then concatenate

pd.DataFrame.append is expensive relative to list.append + a single call of pd.concat.

Therefore, you should aggregate to a list of dataframes and then use pd.concat on this list:

lst = []

for i in segment_dist:
    # do something
    temp = pd.DataFrame(...)
    lst.append(temp)

df = pd.concat(lst, ignore_index=True, axis=0)

df.to_csv(...)

Upvotes: 1

koPytok
koPytok

Reputation: 3713

From df.append documentation:

Append rows of other to the end of this frame, returning a new object. Columns not in this frame are added as new columns.

If you want to add column to the right, use pd.concat with axis=1 (means horizontally):

list_of_dfs = [first_df, second_df, ...]
pd.concat(list_of_dfs, axis=1)

You may want to add parameter ignore_index=True if indexes in dataframes don't match.

Upvotes: 1

Related Questions