CODEWITHSUNDEEP
perltokenize
Sumit
Sumit

Reputation: 671

Incorrect tokenizing of perl's heredoc by PPI

I am using PPI in-order to tokenize a perl file. However the heredoc doesn't seem to be tokenized correctly. I am using the below code to tokenize the file:

my $file_name = shift @ARGV;

use PPI;
use PPI::Dumper;

my $Document = PPI::Document->new($file_name);

my $Dumper = PPI::Dumper->new($Document);

$Dumper->print;

__END__

Below is the perl file getting tokenized:

my $name = 'Foo';
 
my $message = <<'END_MESSAGE';
Dear $name,
 
this is a message I plan to send to you.
 
regards
  the Perl Maven
END_MESSAGE
 
print $message;

I get the following output:

PPI::Document
  PPI::Statement::Variable
    PPI::Token::Word    'my'
    PPI::Token::Whitespace      ' '
    PPI::Token::Symbol      '$name'
    PPI::Token::Whitespace      ' '
    PPI::Token::Operator    '='
    PPI::Token::Whitespace      ' '
    PPI::Token::Quote::Single   ''Foo''
    PPI::Token::Structure   ';'
  PPI::Token::Whitespace    '\n'
  PPI::Token::Whitespace    ' \n'
  PPI::Statement::Variable
    PPI::Token::Word    'my'
    PPI::Token::Whitespace      ' '
    PPI::Token::Symbol      '$message'
    PPI::Token::Whitespace      ' '
    PPI::Token::Operator    '='
    PPI::Token::Whitespace      ' '
    PPI::Token::HereDoc     '<<'END_MESSAGE''
    PPI::Token::Structure   ';'
  PPI::Token::Whitespace    '\n'
  PPI::Token::Whitespace    ' \n'
  PPI::Statement
    PPI::Token::Word    'print'
    PPI::Token::Whitespace      ' '
    PPI::Token::Symbol      '$message'
    PPI::Token::Structure   ';'
  PPI::Token::Whitespace    '\n'

Is there any way so that the entire heredoc's value can be obtained ?

Upvotes: 2

Views: 69

Answers (1)

H&#229;kon H&#230;gland
H&#229;kon H&#230;gland

Reputation: 40778

According to the documentation the content of the here-doc is made available with the heredoc method:

my $Document = PPI::Document->new($file_name);
my @heredoc = $Document->find_first('PPI::Token::HereDoc')->heredoc;
say join "", @heredoc;

Output:

Dear $name,

this is a message I plan to send to you.

regards
  the Perl Maven

Upvotes: 3

Related Questions