CODEWITHSUNDEEP
ocamlocamlyaccocamllexmenhir
user3734280
user3734280

Reputation: 21

expand ocamllex for a bigger lexicon

is there a way to get ocammlex work with more keywords? i've written an interpreter and parser for the german language, which "compiles" german text into latex-pictures for sake of language analysis. it works really fine and is really new in the world of linguistics. thanks to all the developers of ocaml, that you can misuse tools like ocamllex and menhir for such things. but soon i will arrive on the limit of the ocammlex-automaton by the size of the lexicon, that it will say: "automaton to big."

i know the solution with the hashtables. I didnt try already, if that solves the problem coming, but does anybody know, if it is easy to change some type of the limiting integer or the like in the ocamllex-source-code, that I can misuse it without changing my program code? in my opinion this error about the size of the automation is needlessly silly and awkward for further abuse, so I want to ask.

thanks and have a nice day.

Upvotes: 2

Views: 113

Answers (1)

Jeffrey Scofield
Jeffrey Scofield

Reputation: 66823

I applaud your persistence in abusing the tools :-)

I looked through the sources for ocamllex and I see just one place that's checking whether the automaton is getting too big.

lexgen.ml near line 780:

let do_alloc_cell used t =
  let available =
    try Hashtbl.find tag_cells t with Not_found -> Ints.empty in
  try
    Ints.choose (Ints.diff available used)
  with
  | Not_found ->
      temp_pending := false ;
      let n = !next_mem_cell in
      if n >= 255 then raise Memory_overflow ;
      Hashtbl.replace tag_cells t (Ints.add n available) ;
      incr next_mem_cell ;
      n

There's just a mysterious comparison against 255, with no comment explaining any invariants etc. I looked through the code briefly and also looked at the Lexing module. I don't see any non-obvious dependencies on the value 255. So it's possible you could build your own copy of ocamllex with a larger value here. You might try 1023 (one less than power of 2).

You might also just want to break down and use a different tool. I realize this is what others are probably telling you. As I say, I admire your persistence.

Upvotes: 3

Related Questions