Boost.Spirit grammar issue

Question

I am attempting to parse terminfo definitions text file. I am new to Boost.Spirit. I have started with simple grammar that only parses comment lines, empty lines and terminal definitions. As the code comment in the grammar shows, uncommenting the [_val = _1] for definition breaks compilation. Why? Can I fix it?

If I ignore the actual terminfo file, I expect the code below to parse this kind of text:

# comment line

first definition line
  second 
  third line

# another comment line

Code:

#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/qi_eol.hpp>
#include <boost/spirit/include/qi_eoi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/spirit/include/phoenix_object.hpp>
#include <vector>
#include <iostream>
#include <string>

namespace termcxx
{

namespace parser
{

namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
namespace px = boost::phoenix;

//using qi::double_;
using ascii::space;
//using px::ref;
using px::construct;

//using qi::eps;
//using qi::lit;
using qi::_val;
using qi::_1;
using ascii::char_;
using qi::eol;
using qi::eoi;


struct context
{
    int dummy;

    context () = default;
    context (context const &) = default;
    context (std::vector<char> a)
    { }
    context (std::vector<char> a, std::vector<char> b)
    { }
};

} }


BOOST_FUSION_ADAPT_STRUCT(
    termcxx::parser::context,
    (int, dummy))


namespace termcxx
{

namespace parser
{

template <typename Iterator>
struct parser
    : qi::grammar<Iterator, context()>
{
    qi::rule<Iterator, std::vector<char> > comment_line
    = (*space >> '#' >> *(char_ - eol) >> (eol | eoi))[_val = _1]
        ;

    qi::rule<Iterator, std::vector<char> > empty_line
    = (*space >> (eol | eoi))[_val = _1]
        ;

    qi::rule<Iterator, std::vector<char> > def_first_line
    = (+(char_ - eol) >> (eol | eoi))[_val = _1]
        ;

    qi::rule<Iterator, std::vector<char> > def_subsequent_line
    = (+space >> +(char_ - eol) >> (eol | eoi))[_val = _1]
        ;

    qi::rule<Iterator, std::vector<char> > definition
    = (def_first_line >> *def_subsequent_line)//[_val = _1] // Uncommenting the [_val = _1] breaks compilation. Why?
        ;

    qi::rule<Iterator, context()> start
    = (*(comment_line
            | empty_line
            | definition))[_val = construct<context> ()]
        ;

    parser()
        : parser::base_type(start)
    { }
};

template struct parser<std::string::iterator>;

} // namespace parser

} // namespace termcxx

Answer 1

why do you insist on specifying [_val=_1]? It's redundant because the default attribute propagation does this. In fact it hurts, see below

Next, the attribute type of (def_first_line >> *def_subsequent_line) is (apparently) not compatible with std::vector<char>. Perhaps you can

• just use the default attribute propagation (which has enough smarts to just keep appending)
• use raw[] to get the complete matched input
• define BOOST_SPIRIT_ACTIONS_ALLOW_ATTR_COMPAT (I'm not sure this is well supported)

Also,

• you can replace std::vector<char> with std::string when relying on automatic attribute propagation
• you can detect the actual exposed type of a parser expression: Detecting the parameter types in a Spirit semantic action

Update

A few more issues:

• You had mispelled the attribute types for most rules (missing ()):

  qi::rule<Iterator, std::string()> comment_line;
  qi::rule<Iterator, std::string()> empty_line;
  qi::rule<Iterator, std::string()> def_first_line;
  qi::rule<Iterator, std::string()> def_subsequent_line;
  qi::rule<Iterator, std::string()> definition;
  

• the empty_line matched at eoi leading to infinite loop at end of input

• the use of char_ also accepts spaces (use graph instead:)

      def_first_line      = graph >> +(char_ - eol)         >> (eol|eoi);
  

• the use of qi::space also eats line-ends! Use qi::blank instead

• favour reability:

      empty_line          = *blank >> eol;
      comment_line        = *blank >> '#' >> *(char_ - eol) >> (eol|eoi);
      def_first_line      = graph >> +(char_ - eol)         >> (eol|eoi);
      def_subsequent_line = +blank >> +(char_ - eol)        >> (eol|eoi);
  
      definition          = (def_first_line >> *def_subsequent_line);
  
      start               = (  
                              *(comment_line | empty_line | definition)
                            ) [ _val = px::construct<context>() ]
                            ;
  

  This simple habit will save you hours and hours of work and your sanity when working with Spirit.

• You could simplify the includes somewhat

Here's a fixed up version Live On Coliru with output:

<start>
  <try># comment line\n\nfirs</try>
  <comment_line>
    <try># comment line\n\nfirs</try>
    <success>\nfirst definition li</success>
    <attributes>[[ , c, o, m, m, e, n, t,  , l, i, n, e]]</attributes>
  </comment_line>
  <comment_line>
    <try>\nfirst definition li</try>
    <fail/>
  </comment_line>
  <empty_line>
    <try>\nfirst definition li</try>
    <success>first definition lin</success>
    <attributes>[[]]</attributes>
  </empty_line>
  <comment_line>
    <try>first definition lin</try>
    <fail/>
  </comment_line>
  <empty_line>
    <try>first definition lin</try>
    <fail/>
  </empty_line>
  <definition>
    <try>first definition lin</try>
    <def_first_line>
      <try>first definition lin</try>
      <success>  second \n  third li</success>
      <attributes>[[f, i, r, s, t,  , d, e, f, i, n, i, t, i, o, n,  , l, i, n, e]]</attributes>
    </def_first_line>
    <def_subsequent_line>
      <try>  second \n  third li</try>
      <success>  third line\n\n# anot</success>
      <attributes>[[f, i, r, s, t,  , d, e, f, i, n, i, t, i, o, n,  , l, i, n, e,  ,  , s, e, c, o, n, d,  ]]</attributes>
    </def_subsequent_line>
    <def_subsequent_line>
      <try>  third line\n\n# anot</try>
      <success>\n# another comment l</success>
      <attributes>[[f, i, r, s, t,  , d, e, f, i, n, i, t, i, o, n,  , l, i, n, e,  ,  , s, e, c, o, n, d,  ,  ,  , t, h, i, r, d,  , l, i, n, e]]</attributes>
    </def_subsequent_line>
    <def_subsequent_line>
      <try>\n# another comment l</try>
      <fail/>
    </def_subsequent_line>
    <success>\n# another comment l</success>
    <attributes>[[f, i, r, s, t,  , d, e, f, i, n, i, t, i, o, n,  , l, i, n, e,  ,  , s, e, c, o, n, d,  ,  ,  , t, h, i, r, d,  , l, i, n, e]]</attributes>
  </definition>
  <comment_line>
    <try>\n# another comment l</try>
    <fail/>
  </comment_line>
  <empty_line>
    <try>\n# another comment l</try>
    <success># another comment li</success>
    <attributes>[[]]</attributes>
  </empty_line>
  <comment_line>
    <try># another comment li</try>
    <success></success>
    <attributes>[[ , a, n, o, t, h, e, r,  , c, o, m, m, e, n, t,  , l, i, n, e, !]]</attributes>
  </comment_line>
  <comment_line>
    <try></try>
    <fail/>
  </comment_line>
  <empty_line>
    <try></try>
    <fail/>
  </empty_line>
  <definition>
    <try></try>
    <def_first_line>
      <try></try>
      <fail/>
    </def_first_line>
    <fail/>
  </definition>
  <success></success>
  <attributes>[]</attributes>
</start>
Success

Full code for reference:

#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/fusion/include/adapt_struct.hpp>

#include <vector>
#include <iostream>
#include <string>

namespace qi = boost::spirit::qi;

namespace termcxx { namespace parser {

    namespace ascii = boost::spirit::ascii;
    namespace px    = boost::phoenix;

    //using qi::double_;
    using ascii::blank;
    //using px::ref;
    using px::construct;

    //using qi::eps;
    //using qi::lit;
    using qi::_val;
    using qi::_1;
    using ascii::char_;
    using ascii::graph;
    using qi::eol;
    using qi::eoi;

    struct context
    {
        int dummy;

        context () = default;
        context (context const &) = default;
        context (std::vector<char> a) { }
        context (std::vector<char> a, std::vector<char> b) { }
    };

} }

BOOST_FUSION_ADAPT_STRUCT(termcxx::parser::context, (int, dummy))

namespace termcxx { namespace parser {

    template <typename Iterator>
    struct parser : qi::grammar<Iterator, context()>
    {
        parser() : parser::base_type(start)
        { 
            empty_line          = *blank >> eol;
            comment_line        = *blank >> '#' >> *(char_ - eol) >> (eol|eoi);
            def_first_line      = graph >> +(char_ - eol)         >> (eol|eoi);
            def_subsequent_line = +blank >> +(char_ - eol)        >> (eol|eoi);

            definition          = (def_first_line >> *def_subsequent_line);

            start               = (  
                                    *(comment_line | empty_line | definition)
                                  ) [ _val = px::construct<context>() ]
                                  ;

            BOOST_SPIRIT_DEBUG_NODES((start)(def_first_line)(def_subsequent_line)(definition)(empty_line)(comment_line))
        }

      private:
        qi::rule<Iterator, context()> start;
        qi::rule<Iterator, std::string()> comment_line;
        qi::rule<Iterator, std::string()> empty_line;
        qi::rule<Iterator, std::string()> def_first_line;
        qi::rule<Iterator, std::string()> def_subsequent_line;
        qi::rule<Iterator, std::string()> definition;
    };

} }

int main()
{
    using It = boost::spirit::istream_iterator;
    termcxx::parser::parser<It> g;

    It f(std::cin >> std::noskipws), l;
    termcxx::parser::context data;
    if (qi::parse(f,l,g,data))
        std::cout << "Success\n";
    else
        std::cout << "Failure\n";

    if (f != l)
        std::cout << "Remaining input: '" << std::string(f,l) << "'\n";
}

Answer 2

Let's see what exactly happens at this line:

qi::rule<Iterator, std::vector<char> > definition
    = (def_first_line >> *def_subsequent_line)[_val = _1];
        ;

1. def_first_line is a rule. Its attribute is a std::vector<char>.
2. def_subsequent_line is another rule. Again its attribute is a std::vector<char>.
3. * def_subsequent_line is a parser obtained by applying the kleene operator * to the def_subsequent_line. Its implicit attribute is a vector< std::vector<char> >.
4. (def_first_line >> *def_subsequent_line). This is another parser. Due to the spirit compound attribute rules, its implicit attribute is again vector< std::vector<char> >.

So basically, the line should read:

qi::rule<Iterator, std::vector<std::vector<char> > > definition
    = (def_first_line >> *def_subsequent_line)[_val = _1];
        ;

That makes sense, doesn't it? You want to get every line separately, not all the characters together in the same vector.

Now, as side notes:

• [_val = _1] is not really necessary. You should initialize your rules in the grammar's constructor using the operator %=, which takes care of the implicit attributes.
• Assuming that you don't need to access the comments, you should write a skipper rule, which can automatically handle spacing and comments, and then use such rule with phrase_parse.
• You can use std::string in place of vector<char>, spirit is smart enough to understand that a sequence of characters is a string.
• Look here for the compound attribute rules.