CODEWITHSUNDEEP
pythonpython-3.xmatlabexport-to-csvopenmodelica
Matthi9000
Matthi9000

Reputation: 1227

How can I export variables from .mat file (generated by Dymola) to .csv using python

I'm a student who is quite new to coding in Python. I'm using Dymola for several years and now I'm using the Dymola/Python interface with which you can operate Dymola from inside Python (useful for building stock simulations, global sensitivity analysis etc.).

Now, Dymola always generates .mat files in an efficient unreadable data structure. I was wondering how to export variables I'm interested in from that .mat-file to .csv using a Python-script? (I don't want the whole file to be converted to .csv because it is simple way too large)

I'm aware of a DyMat-package for Python that should do the job but either I don't understand the code or the code is not doing what it should do? Does anybody have experience with this? I probably miss some code defining which .mat file has to be read/exported from, which variables I want and in which directory the result.csv-file should be stored?

import csv, numpy

def export(dm, varList, fileName=None, formatOptions={}):
    """Export DyMat data to a CSV file"""

    if not fileName:
        fileName = dm.fileName + '.csv'
    oFile = open(fileName, 'w')
    csvWriter = csv.writer(oFile)

    vDict = dm.sortByBlocks(varList)
    for vList in vDict.values():
        vData = dm.getVarArray(vList)
        vList.insert(0, dm._absc[0])
        csvWriter.writerow(vList)
        csvWriter.writerows(numpy.transpose(vData))

    oFile.close()

Thanks!

Upvotes: 2

Views: 919

Answers (3)

Ragner
Ragner

Reputation: 1

For small files (<2GB) Buildingspy (or other Python packages) covers pretty much all needs: https://simulationresearch.lbl.gov/modelica/buildingspy/

However, since one will run into issues when the files are above 2GB (e.g. for full years of simulations), "alist.exe" from Dymola may be employed. (filterSimulationResults from OpenModelica also fails then)

"alist.exe" seems to accept up until approx. 100 variables to be exported at once and single executions for each variable seems to slow things down drastically (translation of 1 or 100 rows takes almost the same time). One may employ the alist.exe as follows from Python to facilitate automation and speed things up.

var_list=['Component.Name1','Component.Name3','Component.Name2','...'] #List of Variabels to be extracted
N_batch=100 #Number of variables to be extracted from the .mat file at once (max. approx 110)

cmds=[]        #list of commands to be executed batch wise 

for i,var in enumerate(var_list):
    if (i%N_batch == 0) &(i > 0):
        cmds.append(cmd)
        cmd=''
    cmd+=f' -e {var}'#build command
    
cmds.append(cmd)


lst_df=[]       #list of pandas dataframes 
for i,cmd in enumerate(cmds):
    os.system(f'"C:/Program Files/Dymola 2021/bin64/alist.exe" {cmd} {inFile} tmp.csv')
    lst_df.append(pd.read_csv('tmp.csv',index_col=[0]).squeeze("columns"))

df_overall=pd.concat(lst_df,axis=1)
df_overall.to_csv('CompleteCSVFile.csv')#or use .pkl for more efficient writing and reading

It is still not a fast solution, but enables the processing of the date in the first instance. Variable Selection of Dymola should always be exploited first before trying to wrangle around such amounts of data on a local machine.

Hope this helps someone someday!

Upvotes: 0

Dag B
Dag B

Reputation: 759

In the Dymola distribution there is a utility called alist.exe, that allows you to export a number of variables in CSV format.

Another possibility is to convert the MAT file to SDF format, which is a very simple HDF5 interpretation. The HDF5 file is not as compact as the MAT-file, but you can compress the file using ZIP/GZIP/7ZIP to reduce archival storage. There are both MATLAB and Python scripts for reading the SDF format in the Dymola distribution.

Upvotes: 2

sjoelund.se
sjoelund.se

Reputation: 3523

Since this was tagged openmodelica, I am proposing a solution using it:

filterSimulationResults("file.mat", "file.csv", {"x","y","z"}) creates a csv-file with only variables x, y, z (If you think it's still too large, it is possible to resample the file).

Upvotes: 0

Related Questions