user3806824
Reputation: 45
R - appending data frame from JSON in for loop
I've looked at many of the different posts about appending a data frame within a for loop but cannot find a solution to my specific problem. Please read below. Any suggestions would be greatly appreciated!
I'm trying to append a data frame within a for loop without rewriting the entire file. I only want to add the new records each time the loop executes. I'm downloading data from an API. Code to interface with the API and download the data works fine. I use jsonlite to parse the json output and can see the data in the console. Unfortunately, when I try to append the data into a data frame, I get an error message, see below.
Current process:
Libraries used:
library(quantmod)
library(ROAuth)
library(jsonlite)
library(plyr)
Create data frame before for loop:
data_all<-data.frame()
Code to append data at end of loop:
data<-fromJSON(response)
data_all<-rbind(data_all,data)
}
Error message:
Error in (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE, :
row names supplied are of the wrong length
Again, the code to interface with API, download, and parse the data works fine. Here's a sample data set viewed in the console:
$response
$response$`@id`
[1] "8193818b-a60c-462c-9700-f52638a75f49"
$response$elapsedtime
[1] "0"
$response$quotes
$response$quotes$quote
adp_100 adp_200 adp_50 adv_21 adv_30 adv_90 ask ask_time asksz basis
1 32.5810 30.0390 32.2500 5135 4973 6973 0.00 00:00 0 na
2 78.1520 72.2150 77.3350 90738 75752 62908 0.00 00:00 0 na
3 6.1742 7.0495 5.8037 8651 7541 11224 0.0000 00:00 0 na
4 1.5372 1.8598 1.4709 40501 98865 108243 0.0000 00:00 0 na
5 15.7944 17.2267 14.8303 191035 208140 218453 0.00 00:00 0 na
6 17.67 17.22 16.84 263510 245725 264352 18.970 00:00 21 na
7 485.8671 462.3571 479.1363 3702 3545 4654 0.00 00:00 0 na
8 34.681 35.654 33.258 120094 108802 101061 0.000 00:00 0 na
9 2.2274 1.9028 2.2863 863965 812934 976648 2.510 00:00 2 na
10 16.3700 15.4000 16.4300 385475 316978 268737 0.000 00:00 0 na
beta bid bid_time bidsz bidtick chg chg_sign chg_t cl
1 0.6801 0.00 00:00 0 d 0 e na 32.65
2 -0.7521 0.00 00:00 0 d 0 e na 68.28
3 -0.0625 0.0000 00:00 0 d 0 e na 5.9900
4 3.4096 0.0000 00:00 0 d 0 e na 1.4787
5 1.4868 12.15 00:00 1 d 0 e na 16.90
6 0.3458 18.960 00:00 23 d 0 e na 19.180
7 0.6924 0.00 00:00 0 d 0 e na 492.80
8 1.8913 0.000 00:00 0 d 0 e na 34.300
9 0.6425 0.000 00:00 0 d 0 e na 2.550
10 2.0775 0.000 00:00 0 d 0 e na 17.105
contract_size cusip date datetime days_to_expiration
1 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
2 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
3 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
4 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
5 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
6 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
7 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
8 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
9 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
10 na na 2017-03-03 2017-03-03T00:00:00-05:00 na
div divexdate divfreq divpaydt dollar_value eps exch exch_desc hi
1 0.22 20170427 Q 20170515 69329.46 1.68 NASD NASDAQ 32.80
2 0.34 20161228 Q 20170109 3562746.42 0.89 NASD NASDAQ 68.10
3 na na Q na 1919.7966 -6.88 NASD NASDAQ 5.9800
4 na na Q na 83955.2782 -4.13 NASD NASDAQ 1.6000
5 na na Q na 2559591.04 -2.9 NASD NASDAQ 17.40
6 na na Q na 5645067.126 -1.15 NASD NASDAQ 19.205
7 1.05 20170313 Q 20170331 884035.88 15.16 NASD NASDAQ 490.75
8 na na Q na 2481311.782 1.7492 NASD NASDAQ 34.385
9 na na Q na 1359155.880 -0.17 NASD NASDAQ 2.560
10 na na Q na 1951265.457 0.52 NASD NASDAQ 17.220
iad idelta igamma imp_volatility incr_vl irho issue_desc itheta ivega
1 0.88 na na na 1278 na na na na
2 1.36 na na na 6202 na na na na
3 na na na na 100 na na na na
4 na na na na 3583 na na na na
5 na na na na 9589 na na na na
6 na na na na 16732 na na na na
7 4.2 na na na 466 na na na na
8 na na na na 7410 na na na na
9 na na na na 8903 na na na na
10 na na na na 14084 na na na na
last lo name op_delivery op_flag
1 32.80 32.60 AMES NATIONAL CORP. na 0
2 66.71 66.59 ATN INTERNATIONAL INC na 1
3 5.9800 5.7243 ATOMERA INC na 0
4 1.5101 1.4400 ATOSSA GENETICS INC na 0
5 17.15 16.85 ATARA BIOTHERAPEUTICS INC na 1
6 18.960 18.700 ATRICURE INC na 1
7 489.75 489.75 ATRION CORP. na 0
8 33.560 33.505 ASTRONICS CORP na 1
9 2.500 2.460 ANTARES PHARMA INC. na 1
10 17.070 16.870 AIR TRANSPORT SERVICES GROUP, INC. na 1
op_style op_subclass openinterest opn opt_val pchg pchg_sign pcls
1 na na na 32.60 na 0.00 na 32.80
2 na na na 67.91 na 0.00 na 66.71
3 na na na 5.7243 na 0.00 na 5.9800
4 na na na 1.4500 na 0.00 na 1.5101
5 na na na 16.95 na 0.00 na 17.15
6 na na na 19.000 na 0.00 na 18.960
7 na na na 490.75 na 0.00 na 489.75
8 na na na 34.330 na 0.00 na 33.560
9 na na na 2.550 na 0.00 na 2.500
10 na na na 17.100 na 0.00 na 17.070
pe phi plo popn pr_adp_100 pr_adp_200 pr_adp_50 pr_date
1 19.5238 32.95 32.65 32.90 32.5817 30.0207 32.2551 2017-03-02
2 74.9551 70.54 68.03 70.23 781.5260 72.1521 77.4760 2017-03-02
3 0 6.2400 5.7700 5.7700 6.1742 7.0571 5.8094 2017-03-02
4 0 1.4803 1.4200 1.4515 1.5372 1.8732 1.4715 2017-03-02
5 0 17.35 16.25 16.40 15.7944 17.2638 14.8294 2017-03-02
6 0 19.590 18.520 18.880 176.73 17.21 16.82 2017-03-02
7 32.3054 493.30 488.35 488.35 485.8671 462.2427 478.6029 2017-03-02
8 19.1859 35.425 33.980 34.730 346.814 35.661 33.292 2017-03-02
9 0 2.565 2.480 2.510 2.2274 1.8969 2.2805 2017-03-02
10 32.8269 17.340 17.040 17.340 163.7500 15.4012 16.4200 2017-03-02
pr_openinterest prbook prchg prem_mult put_call pvol qcond rootsymbol
1 na 1.79 -0.25 na na 5559 38 na
2 na 1.60 -1.95 na na 49971 38 na
3 na 1.0678 0.2200 na na 4653 38 na
4 na 0.5741 0.0272 na na 22806 38 na
5 na 1.83 0.50 na na 294468 38 na
6 na 3.710 0.300 na na 692148 38 na
7 na 5.58 4.45 na na 3315 38 na
8 na 2.206 -0.430 na na 105349 38 na
9 na 7.812 0.040 na na 569164 38 na
10 na 3.178 -0.235 na na 119506 38 na
secclass sesn sho strikeprice symbol tcond timestamp tr_num tradetick
1 0 na 9311000 na ATLO 29 1488790771 28 e
2 0 na 16144000 na ATNI 29 1488790771 1031 e
3 0 na 5357000 na ATOM 191 1488790771 8 e
4 0 na 3788000 na ATOS 73 1488790771 105 e
5 0 na 28861000 na ATRA 29 1488790771 1476 e
6 0 na 33240000 na ATRC 29 1488790771 2541 e
7 0 na 1824000 na ATRI 29 1488790771 72 e
8 0 na 21692000 na ATRO 29 1488790771 848 e
9 0 na 155087000 na ATRS 29 1488790771 2772 e
10 0 na 59568000 na ATSG 29 1488790771 1108 e
trend under_cusip undersymbol vl volatility12 vwap wk52hi wk52hidate
1 na na na 2116 0.2336 32.76 37.0 20170106
2 na na na 53289 0.2874 66.85 87.80 20170103
3 na na na 329 0.5125 5.8352 10.2000 20160830
4 na na na 54562 0.9774 1.5387 6.40 20160309
5 na na na 150230 0.7351 17.03 25.73 20160805
6 na na na 297923 0.4456 18.948 20.40 20161115
7 na na na 1806 0.2029 489.49 522 20161215
8 na na na 73551 0.4079 33.735 41.00 20161209
9 na na na 544378 0.6475 2.496 2.67 20170215
10 na na na 114571 0.3617 17.031 17.60 20170224
wk52lo wk52lodate xdate xday xmonth xyear yield
1 23.0 20160311 na na na na 2.6829
2 62.70 20160817 na na na na 2.0387
3 5.5000 20170228 na na na na na
4 1.30 20161230 na na na na na
5 12.45 20170131 na na na na na
6 13.44 20160512 na na na na na
7 373 20160323 na na na na 0.8576
8 25.90 20160504 na na na na na
9 0.67 20160318 na na na na na
10 11.70 20160307 na na na na na
$response$error
[1] "Success"
Do I need to process the data by converting the NA values prior to appending? Should I use a different library or function(s) to append the data without rewriting the entire file? Thanks very much for any suggestions.
Upvotes: 0
Views: 1603
Answers (1)
takje
Reputation: 2800
This code is reproducible and appends a JSON to a data.frame.
library(jsonlite)
library(dplyr)
data_all <- data.frame()
response <- toJSON(mtcars, pretty=TRUE)
for(i in 1:3){
data <- fromJSON(response)
data_all <- bind_rows(data_all, data)
}
While data is only 32 rows, data_all has 96 rows.
Upvotes: 3
Related Questions
- Nested JSON to Data Frame
- How to append json data into dataframe and update its value in R?
- For Loop reading API response with existing Data Frame
- json to dataframe
- R loops with JSON API Source
- Adding a list of data frames to one of the variables in a data frame (R)
- Creating a data.frame with a new row for each iteration
- R: Appending to a data frame in a for loop
- Build a data frame in R from a series of JSON files
- Building a data frame row by row from a list