org.antlr.v4.runtime.Token.getTokenSource()方法的使用及代码示例

x33g5p2x  于2022-01-29 转载在 其他  
字(11.8k)|赞(0)|评价(0)|浏览(104)

本文整理了Java中org.antlr.v4.runtime.Token.getTokenSource()方法的一些代码示例，展示了Token.getTokenSource()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Token.getTokenSource()方法的具体详情如下：
包路径：org.antlr.v4.runtime.Token
类名称：Token
方法名：getTokenSource

Token.getTokenSource介绍

[英]Gets the TokenSource which created this token.
[中]

代码示例

代码示例来源：origin: graphql-java/graphql-java

protected SourceLocation getSourceLocation(Token token) {
  String sourceName = token.getTokenSource().getSourceName();
  if (IntStream.UNKNOWN_SOURCE_NAME.equals(sourceName)) {
    // UNKNOWN_SOURCE_NAME is Antrl's way of indicating that no source name was given during parsing --
    // which is the case when queries and other operations are parsed. We don't want this hardcoded
    // '<unknown>' sourceName to leak to clients when the response is serialized as JSON, so we null it.
    sourceName = null;
  }
  return new SourceLocation(token.getLine(), token.getCharPositionInLine() + 1, sourceName);
}

代码示例来源：origin: prestodb/presto

@Override
  public void exitNonReserved(SqlBaseParser.NonReservedContext context)
  {
    // we can't modify the tree during rule enter/exit event handling unless we're dealing with a terminal.
    // Otherwise, ANTLR gets confused an fires spurious notifications.
    if (!(context.getChild(0) instanceof TerminalNode)) {
      int rule = ((ParserRuleContext) context.getChild(0)).getRuleIndex();
      throw new AssertionError("nonReserved can only contain tokens. Found nested rule: " + ruleNames.get(rule));
    }
    // replace nonReserved words with IDENT tokens
    context.getParent().removeLastChild();
    Token token = (Token) context.getChild(0).getPayload();
    context.getParent().addChild(new CommonToken(
        new Pair<>(token.getTokenSource(), token.getInputStream()),
        SqlBaseLexer.IDENTIFIER,
        token.getChannel(),
        token.getStartIndex(),
        token.getStopIndex()));
  }
}

代码示例来源：origin: org.antlr/antlr4-runtime

getTokenFactory().create(new Pair<TokenSource, CharStream>(tok.getTokenSource(), tok.getTokenSource().getInputStream()),
               expectedTokenType, tok.getText(),
               Token.DEFAULT_CHANNEL,
Token tok = e.getOffendingToken();
Token errToken =
  getTokenFactory().create(new Pair<TokenSource, CharStream>(tok.getTokenSource(), tok.getTokenSource().getInputStream()),
               Token.INVALID_TYPE, tok.getText(),
               Token.DEFAULT_CHANNEL,

代码示例来源：origin: org.antlr/antlr4-runtime

/**
 * Constructs a new {@link CommonToken} as a copy of another {@link Token}.
 *
 * <p>
 * If {@code oldToken} is also a {@link CommonToken} instance, the newly
 * constructed token will share a reference to the {@link #text} field and
 * the {@link Pair} stored in {@link #source}. Otherwise, {@link #text} will
 * be assigned the result of calling {@link #getText}, and {@link #source}
 * will be constructed from the result of {@link Token#getTokenSource} and
 * {@link Token#getInputStream}.</p>
 *
 * @param oldToken The token to copy.
 */
public CommonToken(Token oldToken) {
  type = oldToken.getType();
  line = oldToken.getLine();
  index = oldToken.getTokenIndex();
  charPositionInLine = oldToken.getCharPositionInLine();
  channel = oldToken.getChannel();
  start = oldToken.getStartIndex();
  stop = oldToken.getStopIndex();
  if (oldToken instanceof CommonToken) {
    text = ((CommonToken)oldToken).text;
    source = ((CommonToken)oldToken).source;
  }
  else {
    text = oldToken.getText();
    source = new Pair<TokenSource, CharStream>(oldToken.getTokenSource(), oldToken.getInputStream());
  }
}

代码示例来源：origin: org.antlr/antlr4-runtime

recognizer.getTokenFactory().create(new Pair<TokenSource, CharStream>(current.getTokenSource(), current.getTokenSource().getInputStream()), expectedTokenType, tokenText,
        Token.DEFAULT_CHANNEL,
        -1, -1,

代码示例来源：origin: nielsbasjes/yauaa

@Override
public void enterUserAgent(UserAgentContext ctx) {
  // In case of a parse error the 'parsed' version of agent can be incomplete
  inform(ctx, AGENT, ctx.start.getTokenSource().getInputStream().toString());
}

代码示例来源：origin: korpling/ANNIS

public ListTokenSource(List<Token> token)
{
 this.token = token;
 Preconditions.checkNotNull(token);
 Preconditions.checkArgument(!token.isEmpty(), "Internal token list must not be empty");
 
 for(Token t : token)
 {
  if(t.getTokenSource() != null)
  {
   this.factory = t.getTokenSource().getTokenFactory();
   break;
  }
 }
 Preconditions.checkNotNull(this.factory, "Internal token list needs a valid TokenSource");
 
 Token lastToken = token.get(token.size()-1);
 eofToken.setLine(lastToken.getLine());
 eofToken.setCharPositionInLine(lastToken.getCharPositionInLine());
}

代码示例来源：origin: org.ballerinalang/language-server-core

@Override
public void reportUnwantedToken(Parser parser) {
  if (!parser.getContext().start.getTokenSource().getSourceName()
      .equals(context.get(DocumentServiceKeys.RELATIVE_FILE_PATH_KEY).replace("\\", "/"))) {
    return;
  }
  this.context.put(CompletionKeys.TOKEN_STREAM_KEY, parser.getTokenStream());
}

代码示例来源：origin: org.ballerinalang/language-server-core

@Override
public void reportInputMismatch(Parser parser, InputMismatchException e) {
  if (!parser.getContext().start.getTokenSource().getSourceName()
      .equals(context.get(DocumentServiceKeys.RELATIVE_FILE_PATH_KEY).replace("\\", "/"))) {
    return;
  }
  this.context.put(CompletionKeys.TOKEN_STREAM_KEY, parser.getTokenStream());
}

代码示例来源：origin: org.ballerinalang/language-server-core

@Override
public void reportMissingToken(Parser parser) {
  if (!parser.getContext().start.getTokenSource().getSourceName()
      .equals(context.get(DocumentServiceKeys.RELATIVE_FILE_PATH_KEY).replace("\\", "/"))) {
    return;
  }
  this.context.put(CompletionKeys.TOKEN_STREAM_KEY, parser.getTokenStream());
}

代码示例来源：origin: org.ballerinalang/language-server-core

@Override
public void reportNoViableAlternative(Parser parser, NoViableAltException e) {
  if (!parser.getContext().start.getTokenSource().getSourceName()
      .equals(context.get(DocumentServiceKeys.RELATIVE_FILE_PATH_KEY).replace("\\", "/"))) {
    return;
  }
  this.context.put(CompletionKeys.TOKEN_STREAM_KEY, parser.getTokenStream());
}

代码示例来源：origin: com.tunnelvisionlabs/antlr4-runtime

protected Token constructToken(TokenSource tokenSource, int expectedTokenType, String tokenText, Token current) {
  TokenFactory factory = tokenSource.getTokenFactory();
  return
    factory.create(Tuple.create(tokenSource, current.getTokenSource().getInputStream()), expectedTokenType, tokenText,
            Token.DEFAULT_CHANNEL,
            -1, -1,
            current.getLine(), current.getCharPositionInLine());
}

代码示例来源：origin: org.ballerinalang/language-server-core

@Override
public void reportMatch(Parser recognizer) {
  if (!recognizer.getContext().start.getTokenSource().getSourceName()
      .equals(context.get(DocumentServiceKeys.RELATIVE_FILE_PATH_KEY).replace("\\", "/"))) {
    super.reportMatch(recognizer);
    return;
  }
  removePendingTokensAfterThisToken(recognizer, removeStartToken, TokenRemovalStrategy.MATCH);
  boolean inFirstTokenOfCursorLine = isInFirstTokenOfCursorLine(recognizer);
  boolean inLastTermination = isInLastTermination(recognizer);
  super.reportMatch(recognizer);
  if (recognizer.getCurrentToken().getType() != BallerinaParser.EOF && inLastTermination) {
    deleteTokensUpToCursor(recognizer, inLastTermination, inFirstTokenOfCursorLine);
  }
}

代码示例来源：origin: org.ballerinalang/language-server-core

@Override
public void sync(Parser recognizer) throws RecognitionException {
  if (!recognizer.getContext().start.getTokenSource().getSourceName()
      .equals(context.get(DocumentServiceKeys.RELATIVE_FILE_PATH_KEY).replace("\\", "/"))) {
    super.sync(recognizer);
    return;
  }
  removePendingTokensAfterThisToken(recognizer, removeStartToken, TokenRemovalStrategy.SYNC);
  boolean inFirstTokenOfCursorLine = isInFirstTokenOfCursorLine(recognizer);
  boolean inLastTermination = isInLastTermination(recognizer);
  if (recognizer.getCurrentToken().getType() != BallerinaParser.EOF && inFirstTokenOfCursorLine) {
    deleteTokensUpToCursor(recognizer, inLastTermination, inFirstTokenOfCursorLine);
  } else if (recognizer.getCurrentToken().getType() != BallerinaParser.EOF && inLastTermination) {
    deleteTokensUpToCursor(recognizer, inLastTermination, inFirstTokenOfCursorLine);
  }
  super.sync(recognizer);
}

代码示例来源：origin: stanford-futuredata/macrobase

@Override
  public void exitNonReserved(SqlBaseParser.NonReservedContext context) {
    // we can't modify the tree during rule enter/exit event handling unless we're dealing with a terminal.
    // Otherwise, ANTLR gets confused an fires spurious notifications.
    if (!(context.getChild(0) instanceof TerminalNode)) {
      int rule = ((ParserRuleContext) context.getChild(0)).getRuleIndex();
      throw new AssertionError(
        "nonReserved can only contain tokens. Found nested rule: " + ruleNames
          .get(rule));
    }
    // replace nonReserved words with IDENT tokens
    context.getParent().removeLastChild();
    Token token = (Token) context.getChild(0).getPayload();
    context.getParent().addChild(new CommonToken(
      new Pair<>(token.getTokenSource(), token.getInputStream()),
      SqlBaseLexer.IDENTIFIER,
      token.getChannel(),
      token.getStartIndex(),
      token.getStopIndex()));
  }
}

代码示例来源：origin: com.facebook.presto/presto-parser

@Override
  public void exitNonReserved(SqlBaseParser.NonReservedContext context)
  {
    // we can't modify the tree during rule enter/exit event handling unless we're dealing with a terminal.
    // Otherwise, ANTLR gets confused an fires spurious notifications.
    if (!(context.getChild(0) instanceof TerminalNode)) {
      int rule = ((ParserRuleContext) context.getChild(0)).getRuleIndex();
      throw new AssertionError("nonReserved can only contain tokens. Found nested rule: " + ruleNames.get(rule));
    }
    // replace nonReserved words with IDENT tokens
    context.getParent().removeLastChild();
    Token token = (Token) context.getChild(0).getPayload();
    context.getParent().addChild(new CommonToken(
        new Pair<>(token.getTokenSource(), token.getInputStream()),
        SqlBaseLexer.IDENTIFIER,
        token.getChannel(),
        token.getStartIndex(),
        token.getStopIndex()));
  }
}

代码示例来源：origin: io.prestosql/presto-parser

@Override
  public void exitNonReserved(SqlBaseParser.NonReservedContext context)
  {
    // we can't modify the tree during rule enter/exit event handling unless we're dealing with a terminal.
    // Otherwise, ANTLR gets confused an fires spurious notifications.
    if (!(context.getChild(0) instanceof TerminalNode)) {
      int rule = ((ParserRuleContext) context.getChild(0)).getRuleIndex();
      throw new AssertionError("nonReserved can only contain tokens. Found nested rule: " + ruleNames.get(rule));
    }
    // replace nonReserved words with IDENT tokens
    context.getParent().removeLastChild();
    Token token = (Token) context.getChild(0).getPayload();
    context.getParent().addChild(new CommonToken(
        new Pair<>(token.getTokenSource(), token.getInputStream()),
        SqlBaseLexer.IDENTIFIER,
        token.getChannel(),
        token.getStartIndex(),
        token.getStopIndex()));
  }
}

代码示例来源：origin: com.yahoo.bullet/bullet-bql

@Override
  public void exitNonReserved(BQLBaseParser.NonReservedContext context) {
    // We can't modify the tree during rule enter/exit event handling unless we're dealing with a terminal.
    // Otherwise, ANTLR gets confused an fires spurious notifications.
    if (!(context.getChild(0) instanceof TerminalNode)) {
      int rule = ((ParserRuleContext) context.getChild(0)).getRuleIndex();
      throw new AssertionError("NonReserved can only contain tokens. Found nested rule: " + ruleNames.get(rule));
    }
    // Replace nonReserved words with IDENT tokens.
    context.getParent().removeLastChild();
    Token token = (Token) context.getChild(0).getPayload();
    context.getParent().addChild(new CommonToken(
        new Pair<>(token.getTokenSource(), token.getInputStream()),
        BQLBaseLexer.IDENTIFIER,
        token.getChannel(),
        token.getStartIndex(),
        token.getStopIndex()));
  }
}

代码示例来源：origin: uk.co.nichesolutions.presto/presto-parser

@Override
public void exitQuotedIdentifier(SqlBaseParser.QuotedIdentifierContext context)
{
  // Remove quotes
  context.getParent().removeLastChild();
  Token token = (Token) context.getChild(0).getPayload();
  context.getParent().addChild(new CommonToken(
      new Pair<>(token.getTokenSource(), token.getInputStream()),
      SqlBaseLexer.IDENTIFIER,
      token.getChannel(),
      token.getStartIndex() + 1,
      token.getStopIndex() - 1));
}

代码示例来源：origin: uk.co.nichesolutions.presto/presto-parser

@Override
  public void exitNonReserved(SqlBaseParser.NonReservedContext context)
  {
    // replace nonReserved words with IDENT tokens
    context.getParent().removeLastChild();
    Token token = (Token) context.getChild(0).getPayload();
    context.getParent().addChild(new CommonToken(
        new Pair<>(token.getTokenSource(), token.getInputStream()),
        SqlBaseLexer.IDENTIFIER,
        token.getChannel(),
        token.getStartIndex(),
        token.getStopIndex()));
  }
}

内容来源于网络，如有侵权，请联系作者删除！

token

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com