CODEWITHSUNDEEP
pythonpython-3.xckan
Matheus
Matheus

Reputation: 150

Load CKAN dataset

I have a few doubt about ckan:

How to:

  1. load a CKAN dataset from web
  2. transform this dataset into a pandas dataframe

And i need have a register in ckan website to query the data?

I am using Pyhton 3.6.1

Edit 2: I had tried the follow code:

 import urllib
url = 'http://dados.cvm.gov.br/api/action/datastore_search?resource_id=92741280-58fc-446b-b436-931faaca4fb4&q=CNPJ_FUNDO:11.286.399/0001-35'
fileobj = urllib.request.urlopen(url)
print(fileobj.read())

But, the result are like this:

b'{"help": "http://dados.cvm.gov.br/api/3/action/help_show?name=datastore_search", "success": true, "result": {"resource_id": "92741280-58fc-446b-b436-931faaca4fb4", "fields": [{"type": "int4", "id": "_id"}, {"type": "text", "id": "CNPJ_FUNDO"}, {"type": "timestamp", "id": "DT_COMPTC"}, {"type": "numeric", "id": "VL_TOTAL"}, {"type": "numeric", "id": "VL_QUOTA"}, {"type": "numeric", "id": "VL_PATRIM_LIQ"}, {"type": "numeric", "id": "CAPTC_DIA"}, {"type": "numeric", "id": "RESG_DIA"}, {"type": "numeric", "id": "NR_COTST"}, {"type": "int8", "id": "_full_count"}, {"type": "float4", "id": "rank"}], "q": "CNPJ_FUNDO:11.286.399/0001-35", "records": [], "_links": {"start": "/api/action/datastore_search?q=CNPJ_FUNDO%3A11.286.399%2F0001-35&resource_id=92741280-58fc-446b-b436-931faaca4fb4", "next": "/api/action/datastore_search?q=CNPJ_FUNDO%3A11.286.399%2F0001-35&offset=100&resource_id=92741280-58fc-446b-b436-931faaca4fb4"}}}'

I need a result like this image

Upvotes: 1

Views: 1188

Answers (1)

Paulo Scardine
Paulo Scardine

Reputation: 77301

  1. load a CKAN dataset from web

The website you linked has a Python example in the link "API de Dados":

import urllib
url = 'http://dados.cvm.gov.br/api/action/datastore_search?resource_id=92741280-58fc-446b-b436-931faaca4fb4&limit=5&q=title:jones'
fileobj = urllib.urlopen(url)
print fileobj.read()
  1. transform this dataset into a pandas dataframe

Do as you do with any JSON dataset, parse it and load in a data frame (there is nothing specific to ckan here):

>>> import pandas as pd
>>> import json
>>> response = json.loads(fileobj.read())
>>> pd.DataFrame(response['result']['records'])

  CAPTC_DIA          CNPJ_FUNDO            DT_COMPTC NR_COTST RESG_DIA  \
0      0.00  00.017.024/0001-53  2017-07-03T00:00:00        1     0.00   
1      0.00  00.017.024/0001-53  2017-07-04T00:00:00        1     0.00   
2      0.00  00.017.024/0001-53  2017-07-05T00:00:00        1     0.00   
3      0.00  00.017.024/0001-53  2017-07-06T00:00:00        1     0.00   
4      0.00  00.017.024/0001-53  2017-07-07T00:00:00        1     0.00   

  VL_PATRIM_LIQ         VL_QUOTA    VL_TOTAL  _id  
0    1111752.99  25.249352000000  1111831.24    1  
1    1112087.29  25.256944400000  1112268.26    2  
2    1112415.28  25.264393500000  1112716.06    3  
3    1112754.06  25.272087600000  1113165.75    4  
4    1113096.62  25.279867600000  1113293.06    5

And i need have a register in ckan website to query the data?

You don't need to register at the website you link, I was able to retrieve the data without registering. I prefer to use the requests library:

import requests
import pandas as pd

params = params={
    'resource_id': '92741280-58fc-446b-b436-931faaca4fb4', 
    'limit': 5,
}
url = 'http://dados.cvm.gov.br/api/action/datastore_search'
r = requests.get(url, params=params).json()

df = pd.DataFrame(r['result']['records'])

Looks like the limit and offset parameters probably behave like in SQL. You may have to convert columns to numeric/date types, again this is nothing specific to ckan and you can find answers on how to do that in the pandas documentation.

>>> df.describe()
            _id
count  5.000000
mean   3.000000
std    1.581139
min    1.000000
25%    2.000000
50%    3.000000
75%    4.000000
max    5.000000

Converting is easy enough:

>>> for col in ('CAPTC_DIA', 'NR_COTST', 'RESG_DIA', 'VL_PATRIM_LIQ', 'VL_QUOTA', 'VL_TOTAL'):
...    df[col] = pd.to_numeric(df[col])

>>> df['DT_COMPTC'] = pd.to_datetime(df['DT_COMPTC'])

>>> df.describe()
       CAPTC_DIA  NR_COTST  RESG_DIA  VL_PATRIM_LIQ   VL_QUOTA      VL_TOTAL  \
count        5.0       5.0       5.0   5.000000e+00   5.000000  5.000000e+00
mean         0.0       1.0       0.0   1.112421e+06  25.264529  1.112655e+06
std          0.0       0.0       0.0   5.303356e+02   0.012045  6.123444e+02
min          0.0       1.0       0.0   1.111753e+06  25.249352  1.111831e+06
25%          0.0       1.0       0.0   1.112087e+06  25.256944  1.112268e+06
50%          0.0       1.0       0.0   1.112415e+06  25.264394  1.112716e+06
75%          0.0       1.0       0.0   1.112754e+06  25.272088  1.113166e+06
max          0.0       1.0       0.0   1.113097e+06  25.279868  1.113293e+06

            _id  
count  5.000000  
mean   3.000000  
std    1.581139  
min    1.000000  
25%    2.000000  
50%    3.000000  
75%    4.000000  
max    5.000000

Upvotes: 1

Related Questions