CODEWITHSUNDEEP
objective-cparsekit
RudolphProd
RudolphProd

Reputation: 31

How to filter out unwanted tokens using ParseKit?

I am using PK to try and tokenize the .gift file format. I am trying to do the following:

Given a string that is similar to this: @"= 2 + 2"

I am trying to return '2 + 2' without going through the hassle of determining if the token that is going over that string is equal to a symbol and then defining how the output string should be. What I'm trying to do is say that if [PKToken.stringValue isEqualToString: @"="] to pop that value off of the PKTokenizer and then return the rest of the string with formatting still in tact.

Let me know if this was clear enough...

--Skylar.

Upvotes: 3

Views: 176

Answers (2)

Todd Ditchendorf
Todd Ditchendorf

Reputation: 11337

Developer of ParseKit here. Here's my ParseKit Tokenizer Answer.

First, I must say that simple filtering of individual chars is probably better accomplished using Regular Expressions than ParseKit.

That said, if you are trying to do this sort of thing with the ParseKit tokenizer, here's how:

NSString *s = @"= 2 + 2";
PKTokenizer *t = [PKTokenizer tokenizerWithString:s];
t.whitespaceState.reportsWhitespaceTokens = YES;

PKToken *eof = [PKToken EOFToken];
PKToken *tok = nil;

NSMutableArray *toks = [NSMutableArray array];
while ((tok = [t nextToken]) != eof) {
    if (![tok.stringValue isEqualToString:@"="]) {
        [toks addObject:tok];
    }
}

NSString *result = [toks componentsJoinedByString:@""];
NSLog(@"%@", result);

Upvotes: 2

Todd Ditchendorf
Todd Ditchendorf

Reputation: 11337

Developer of ParseKit here. Here's my ParseKit Grammar Answer.

Again, I must say that simple filtering of individual chars is probably better accomplished using Regular Expressions than ParseKit.

That said, if you are trying to do this sort of thing with a ParseKit grammar, here's one way:

My grammar:

@reportsWhitespaceTokens = YES;

@start = (reject! | passThru)+;
reject = '=';
passThru = ~reject;

That ! means discard this token. The ~ is Logical Not.

Define this Assembler callback:

- (void)parser:(PKParser *)p didMatchPassThru:(PKAssembly *)a {
    NSLog(@"%s %@", __PRETTY_FUNCTION__, a);
    PKToken *tok = [a pop];
    if (!a.target) {
        a.target = [NSMutableArray array];
    }
    [a.target addObject:tok];
}

You can see, we simply accumulate the passThru tokens in an array stored as the assembly's target.

My driver code:

NSString *g = // fetch grammar above
PKParser *p = [[PKParserFactory factory] parserFromGrammar:g assembler:self];
NSString *s = @"= 2 + 2";

NSArray *toks = [p parse:s];
NSString *result = [toks componentsJoinedByString:@""];
NSLog(@"res '%@'", result);
NSAssert([result isEqualToString:@"2 + 2"], @"");

At the end there, we just fetch the accumulated tokens (the assembly's target) and combine them to a string.

Upvotes: 2

Related Questions