Parsing xml data with R. The code is very slow. How to make it faster?

Question

I have a very big dataset (about 20 GB) in xml format and I need to parse it into tidy tables to make some statistics. I want to save:
- attribute tag for tag controlfield
- attribute tag for tag datafield
- attribute code for tag subfield
- values that correspond to all these attributes/tags

Folowing example from here (R XML - combining parent and child nodes into data frame) I wrote my own script. It works correctly but it is very very slow... I have more than 1 million records so the script is not practical but I don't know how to optimize it.
Is there a way to improve the performance of my code or is there another way to solve the problem. For me it is not important what package to use (xml2 or xml or some other). What I really need is to convert my data into tidy tables as fast as possible. Here is my data, some comments to data and the code.

Thank you for all possible help.

Data

library(xml2)
library(tidyverse)
df <- read_xml(
'

  1
  20180109004353.0
  
    9780821812419
    print
  
  
    9780821899854
    online
  
  
    DOI
    10.1090/memo/0041
  
  
    SPIRESTeX
    Wong:1961af
  
  
    MSNET
    0383311
  
  
    ZBLATT
    0124.13401
  
  
    INSPIRE
    Math and Math Physics
  
  
    1-112
    Mem.Am.Math.Soc.
    41
    1961
    1214590
  
  
    SPIRES-60
  
  
    Book
  
  
    Citeable
  
  
    HEP
  
  
    CURATOR
    1
    Ann.Math.,43,161
  
  
    CURATOR
    2
    Am.Math.Mon.,53,136
  
  
    CURATOR
    E. Cartan
    Paris
    3
    Lecons sur la theorie des spineurs I
    1938
  
  
    CURATOR
    4
    Enseign.Math.,40,26
  
  
    CURATOR
    Annals Math.,47,510
  
  
    CURATOR
    6
    Comment.Math.Helv.,15,358
  
  
    CURATOR
    9
    Trans.Am.Math.Soc.,61,193
  
  
    CURATOR
    10
    Math.Ann.,88,1
  
  
    CURATOR
    12
    Math.Ann.,60,546
  
  
    CURATOR
    14
    Comment.Math.Helv.,21,261
  
  
    CURATOR
    17
    Abh.Math.Sem.Univ.Hamburg,1,1
  
  
    CURATOR
    18
    Am.J.Math.,67,472
  
  
    CURATOR
    20
    Comment.Math.Helv.,8,3
  
  
    CURATOR
    21
    Tohoku Math.J.,3,322
  
  
    oai:inspirehep.net:1
    INSPIRE:HEP
  
  
    2001-02-23
    2018-01-09
  
  
    Isoclinic N planes in Euclidean 2N space, Clifford parallels in elliptic (2N-1) space, and the Hurwitz matrix equations
  
  
    arXiv
    Isoclinic n-planes in Euclidean 2n-space, Clifford parallels in elliptic (2n-1)-space, and the Hurwitz matrix equations
  
  
    Wong, Yung-Chow
    Hong Kong U.
    Y.C.Wong.1
    0
    908370
  
  
    1961
  
  
    121
  
  
    Second printing with corrections 1971 (ISBN: 0-8218-1241-6).
  
  
    Introduction Part I. Isoclinic nn-planes in E2nE2n and Clifford parallel
  



  17
  20190325225547.0
  
    SPIRESTeX
    Breymayer:1963bh
  
  
    SLAC-TN-63-023
  
  
    Breymayer, K.E.
    SLAC
    K.E.Breymayer.1
    0
    903206
  
  
    SYSTEM INTERCONNECTION AND PROTECTION OF THE MODULATOR
  
  
    1963-03
  
  
    21
  
  
    INSPIRE
  
  
    Mallory, Kenneth B.
    SLAC
    K.B.Mallory.1
    0
    903206
  
  
    SLAC
  
  
    http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=slac-tn-63-023
    slac-tn-63-023
    SLAC
  
  
    1903536
    http://inspirehep.net/record/17/files/slac-tn-63-023.pdf
  
  
    oai:inspirehep.net:17
    INSPIRE:HEP
  
  
    1963-03-01
    2019-03-25
  
  
    SPIRES-329
  
  
    HEP
  
  
    0-0-0-0-0-0-1
    2014-08-13 15:34:53
    Invenio/1.1.2.1260-aa76f refextract/1.5.44
    content.pdf;1
  
  
    will open, thus o~nlng the ac cQxuit to the -
    1
  
  
    2
  
  
    Apply the trigger. If the high voltage is present, the klystron is pulsed and draws average current
    3
    relay comes on. In addI.tion, a light at the modulator comes on, indicating Klystron Current
  
  
    no recycling
    3
  
  
    removes the 260-6~0 vous l (2) Resetting: close circuit breaker only, no further action necessary if fault is cleared.
    3
  
  
    % u --L--_ I ‘2 Q-4 -r---.‘-i-’I ___----_.--_.-_. -.L.-
    4
  



  18
  20140813111316.0
  
    Johnston:1977saa
    INSPIRETeX
  
  
    SLAC-PUB-0341
  
  
    Johnston, Lawrence II.
    SLAC
    L.H.Johnston.1
    0
    903206
  
  
    A target beam monitor using secondary electron emission
  
  
    1977-08
  
  
    17
  
  
    Murray, Joseph J.
    SLAC
    J.J.Murray.1
    0
    903206
  
  
    Gearhart, Roger A.
    LBL, Berkeley
    R.Gearhart.1
    0
    902953
  
  
    Hauser, Todd
    LBL, Berkeley
    T.Hauser.2
    0
    902953
  
  
    Klein, Paul R.
    Purdue U.
    P.Klein.2
    0
    903142
  
  
    0341
    SLACPUB
    http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=SLAC-PUB-0341
  
  
    1110230
    http://inspirehep.net/record/18/files/slac-pub-0341.pdf
  
  
    oai:inspirehep.net:18
    INSPIRE:HEP
  
  
    1977-01-01
    2014-08-13
  
  
    SPIRES-337
  
  
    HEP
  
  
    Rev.Sci.Instrum.,26,229
  
  
    IEEE Trans.Nucl.Sci.,12,935
    48647
  
  
    Nucl.Instrum.Meth.,15,63
  
  
    Phys.Rev.Lett.,18,366
  

')

Comments to tags:
This is identifier of the record
This is time of last record update

My code

start_t <- Sys.time()
x1 <- df %>% xml_find_all('.//record')

x2 <- x1 %>% 
    xml_find_all('.//datafield') %>% 
    map_df(~list(rec      = list(map(xml_find_all(.x, './/preceding-sibling::controlfield'), xml_text)),
                 attrs    = list(xml_attr(.x, attr = 'tag')), 
                 variable = list(map(xml_children(.x), xml_attrs)), 
                 value    = list(map(xml_children(.x), xml_text))
                 ))

x2_tidy <- x2 %>% unnest(c(variable, value)) %>% mutate(rec = as.character(rec))
Sys.time() - start_t

Update My result is like this. Sorry, I can't figure out how to add text table here, so I add it like an image. Image with resulting table

Parsing xml data with R. The code is very slow. How to make it faster?

Answers (1)

Related Questions