本文整理了Java中org.apache.lucene.document.Field.isTokenized()方法的一些代码示例，展示了Field.isTokenized()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Field.isTokenized()方法的具体详情如下：
包路径：org.apache.lucene.document.Field
类名称：Field
方法名：isTokenized

Field.isTokenized介绍

[英]True iff the value of the field should be tokenized as text prior to indexing. Un-tokenized fields are indexed as a single word and may not be Reader-valued.
[中]如果为True，则字段的值应在索引之前标记为文本。未标记的字段作为单个单词编制索引，并且可能不是读取器值。

代码示例

代码示例来源：origin: org.apache.lucene/lucene-fast-vector-highlighter

protected String getFragmentSource( StringBuilder buffer, int[] index, Field[] values,
  int startOffset, int endOffset ){
 while( buffer.length() < endOffset && index[0] < values.length ){
  buffer.append( values[index[0]].stringValue() );
  if( values[index[0]].isTokenized() && values[index[0]].stringValue().length() > 0 && index[0] + 1 < values.length )
   buffer.append( ' ' );
  index[0]++;
 }
 int eo = buffer.length() < endOffset ? buffer.length() : endOffset;
 return buffer.substring( startOffset, eo );
}

代码示例来源：origin: org.sonatype.nexus/nexus-indexer

public void writeField( Field field )
  throws IOException
{
  int flags = ( field.isIndexed() ? F_INDEXED : 0 ) //
    + ( field.isTokenized() ? F_TOKENIZED : 0 ) //
    + ( field.isStored() ? F_STORED : 0 ) //
    + ( field.isCompressed() ? F_COMPRESSED : 0 );
  String name = field.name();
  String value = field.stringValue();
  dos.write( flags );
  dos.writeUTF( name );
  writeUTF( value, dos );
}

代码示例来源：origin: lucene/lucene

final void addDocument(Document doc) throws IOException {
  indexStream.writeLong(fieldsStream.getFilePointer());
  
  int storedCount = 0;
  Enumeration fields  = doc.fields();
  while (fields.hasMoreElements()) {
   Field field = (Field)fields.nextElement();
   if (field.isStored())
  storedCount++;
  }
  fieldsStream.writeVInt(storedCount);
  
  fields  = doc.fields();
  while (fields.hasMoreElements()) {
   Field field = (Field)fields.nextElement();
   if (field.isStored()) {
  fieldsStream.writeVInt(fieldInfos.fieldNumber(field.name()));

  byte bits = 0;
  if (field.isTokenized())
   bits |= 1;
  fieldsStream.writeByte(bits);

  fieldsStream.writeString(field.stringValue());
   }
  }
 }
}

代码示例来源：origin: lucene/lucene

if (!field.isTokenized()) {		  // un-tokenized field
 addPosition(fieldName, field.stringValue(), position++);
 length++;