cascading.tuple.Tuple类的使用及代码示例

x33g5p2x  于2022-01-29 转载在 其他  
字(11.8k)|赞(0)|评价(0)|浏览(110)

本文整理了Java中cascading.tuple.Tuple类的一些代码示例,展示了Tuple类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Tuple类的具体详情如下:
包路径:cascading.tuple.Tuple
类名称:Tuple

Tuple介绍

[英]A Tuple represents a set of values. Consider a Tuple the same as a database record where every value is a column in that table.

A "tuple stream" is a set of Tuple instances passed consecutively through a Pipe assembly.

Tuples work in tandem with Fields and the TupleEntry classes. A TupleEntry holds an instance of Fields and a Tuple. It allows a tuple to be accessed by its field names, and will help maintain consistent types if any are given on the Fields instance. That is, if a field is specified at an Integer, calling #set(int,Object)with a String will force the String to be coerced into a Integer instance.

For managing custom types, see the CoercibleType interface which extends Type.

Tuple instances created by user code, by default, are mutable (or modifiable). Tuple instances created by the system are immutable (or unmodifiable, tested by calling #isUnmodifiable()).

For example tuples returned by cascading.operation.FunctionCall#getArguments(), will always be unmodifiable. Thus they must be copied if they will be changed by user code or cached in the local context. See the Tuple copy constructor, or *Copy() methods on TupleEntry.

Because a Tuple can hold any Object type, it is suitable for storing custom types. But all custom types must have a serialization support per the underlying framework.

For Hadoop, a org.apache.hadoop.io.serializer.Serialization implementation must be registered with Hadoop. For further performance improvements, see the cascading.tuple.hadoop.SerializationToken Java annotation.
[中]元组表示一组值。考虑一个元组与数据库记录相同,其中每个值是该表中的一列。
“元组流”是通过管道组件连续传递的一组元组实例。
元组与字段和TupleEntry类协同工作。TupleEntry包含字段和元组的实例。它允许通过字段名访问元组,并将帮助维护字段实例上给定的一致类型。也就是说,如果在整数处指定了字段,则使用字符串调用#set(int,Object)将强制将该字符串强制转换为整数实例。
有关管理自定义类型的信息,请参见扩展类型的强制类型接口。
默认情况下,由用户代码创建的元组实例是可变的(或可修改的)。系统创建的元组实例是不可变的(或不可修改的,通过调用#isUnmodifiable()进行测试)。
例如,通过级联返回的元组。活动FunctionCall#getArguments()始终不可修改。因此,如果它们将被用户代码更改或缓存在本地上下文中,则必须复制它们。请参阅Tuple copy构造函数,或TupleEntry上的*copy()方法。
因为元组可以保存任何对象类型,所以它适合存储自定义类型。但所有自定义类型都必须根据底层框架具有序列化支持。
对于Hadoop,一个组织。阿帕奇。hadoop。伊奥。序列化程序。序列化实现必须在Hadoop中注册。有关进一步的性能改进,请参阅级联。元组。hadoop。SerializationToken Java注释。

代码示例

代码示例来源:origin: LiveRamp/cascading_ext

@Override
public Tuple toTuple(Number[] values) {
 Tuple tuple = new Tuple();
 for (Number value : values) {
  tuple.add(value);
 }
 return tuple;
}

代码示例来源:origin: elastic/elasticsearch-hadoop

@Override
  public void convert(Object from, BytesArray to) {
    // expect a tuple holding one field - chararray or bytearray
    Assert.isTrue(from instanceof SinkCall,
        String.format("Unexpected object type, expecting [%s], given [%s]", SinkCall.class, from.getClass()));

    // handle common cases
    SinkCall sinkCall = (SinkCall) from;
    Tuple rawTuple = sinkCall.getOutgoingEntry().getTuple();

    if (rawTuple == null || rawTuple.isEmpty()) {
      to.bytes("{}");
      return;
    }
    Assert.isTrue(rawTuple.size() == 1, "When using JSON input, only one field is expected");

    // postpone the coercion
    Tuple tuple = CascadingUtils.coerceToString(sinkCall);
    super.convert(tuple.getObject(0), to);
  }
}

代码示例来源:origin: cwensel/cascading

Tuple nulledCopy( int[] pos )
 {
 if( pos == null )
  return size( size() );
 Tuple results = new Tuple( this );
 for( int i : pos )
  results.set( i, null );
 return results;
 }

代码示例来源:origin: elastic/elasticsearch-hadoop

@SuppressWarnings("unchecked")
@Override
public Result write(SinkCall<Object[], ?> sinkCall, Generator generator) {
  Tuple tuple = CascadingUtils.coerceToString(sinkCall);
  // consider names (in case of aliases these are already applied)
  List<String> names = (List<String>) sinkCall.getContext()[SINK_CTX_ALIASES];
  generator.writeBeginObject();
  for (int i = 0; i < tuple.size(); i++) {
    String name = (i < names.size() ? names.get(i) : "tuple" + i);
    // filter out fields
    if (shouldKeep(generator.getParentPath(), name)) {
      generator.writeFieldName(name);
      Object object = tuple.getObject(i);
      Result result = jdkWriter.write(object, generator);
      if (!result.isSuccesful()) {
        if (object instanceof Writable) {
          return writableWriter.write((Writable) object, generator);
        }
        return Result.FAILED(object);
      }
    }
  }
  generator.writeEndObject();
  return Result.SUCCESFUL();
}

代码示例来源:origin: cwensel/cascading

@Test
public void testGeneratorAggregator() throws Exception
 {
 getPlatform().copyFromLocal( inputFileApache );
 Tap source = getPlatform().getTextFile( new Fields( "offset", "line" ), inputFileApache );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Fields( "line" ), new RegexParser( new Fields( "ip" ), "^[^ ]*" ), new Fields( "ip" ) );
 pipe = new GroupBy( pipe, new Fields( "ip" ) );
 pipe = new Every( pipe, new TestAggregator( new Fields( "count1" ), new Fields( "ip" ), new Tuple( "first1" ), new Tuple( "first2" ) ) );
 pipe = new Every( pipe, new TestAggregator( new Fields( "count2" ), new Fields( "ip" ), new Tuple( "second" ), new Tuple( "second2" ), new Tuple( "second3" ) ) );
 Tap sink = getPlatform().getTextFile( getOutputPath( "generatoraggregator" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 validateLength( flow, 8 * 2 * 3, null );
 }

代码示例来源:origin: cwensel/cascading

@Test(expected = OperationException.class)
public void testGetMissingFail() throws Exception
 {
 TupleEntry entry = new TupleEntry( new Fields( "json", JSONCoercibleType.TYPE ), Tuple.size( 1 ) );
 entry.setObject( 0, JSONData.nested );
 JSONGetFunction function = new JSONGetFunction( new Fields( "result" ), true, "/person/foobar" );
 invokeFunction( function, entry, new Fields( "result" ) );
 }

代码示例来源:origin: cwensel/cascading

void runTestCount( String name, Fields argumentSelector, Fields fieldDeclaration, Fields outputSelector ) throws Exception
 {
 getPlatform().copyFromLocal( inputFileIps );
 Tap source = getPlatform().getTextFile( Fields.size( 2 ), inputFileIps );
 Tap sink = getPlatform().getTextFile( Fields.size( 1 ), getOutputPath( name ), SinkMode.REPLACE );
 Pipe pipe = new Pipe( "count" );
 pipe = new GroupBy( pipe, new Fields( 1 ) );
 pipe = new Every( pipe, argumentSelector, new Count( fieldDeclaration ), outputSelector );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.start(); // simple test for start
 flow.complete();
 validateLength( flow, 17 );
 assertTrue( getSinkAsList( flow ).contains( new Tuple( "63.123.238.8\t2" ) ) );
 }

代码示例来源:origin: cwensel/cascading

@Test
public void testSelect()
 {
 Fields selector = new Fields( "a", "d" );
 TupleEntry entryA = new TupleEntry( new Fields( "a", "b" ), new Tuple( "a", "b" ) );
 TupleEntry entryB = new TupleEntry( new Fields( "c", "d" ), new Tuple( "c", "d" ) );
 Tuple tuple = TupleEntry.select( selector, entryA, entryB );
 assertEquals( "wrong size", 2, tuple.size() );
 assertEquals( "not equal: tuple.get(0)", "a", tuple.getObject( 0 ) );
 assertEquals( "not equal: tuple.get(1)", "d", tuple.getObject( 1 ) );
 }

代码示例来源:origin: cwensel/cascading

private TupleEntry getEntry( Comparable lhs, Comparable rhs )
 {
 Fields fields = new Fields( "a", "b" );
 Tuple parameters = new Tuple( lhs, rhs );
 return new TupleEntry( fields, parameters );
 }

代码示例来源:origin: cwensel/cascading

@Override
public void doAssert( FlowProcess flowProcess, ValueAssertionCall assertionCall )
 {
 TupleEntry input = assertionCall.getArguments();
 int pos = 0;
 for( Object element : input.getTuple() )
  {
  if( !value.equals( element ) )
   fail( input.getFields().get( pos ), element, value, input.getTuple().print() );
  pos++;
  }
 }

代码示例来源:origin: cwensel/cascading

@Test
public void testSet()
 {
 TupleEntry entryA = new TupleEntry( new Fields( "a", "b", "c" ), new Tuple( "a", "b", "c" ) );
 TupleEntry entryB = new TupleEntry( new Fields( "c", "b" ), new Tuple( "C", "B" ) );
 entryA.set( entryB );
 Tuple tuple = entryA.getTuple();
 assertEquals( "wrong size", 3, tuple.size() );
 assertEquals( "not equal: tuple.get(0)", "a", tuple.getObject( 0 ) );
 assertEquals( "not equal: tuple.get(1)", "B", tuple.getObject( 1 ) );
 assertEquals( "not equal: tuple.get(2)", "C", tuple.getObject( 2 ) );
 }

代码示例来源:origin: cwensel/cascading

@Test
public void testWildcard()
 {
 Fields aFields = Fields.ALL;
 Tuple aTuple = tuple.get( fields, aFields );
 assertEquals( "not equal: aTuple.size()", 5, aTuple.size() );
 assertEquals( "not equal: aTuple.get( 0 )", "a", aTuple.get( fields, new Fields( "one" ) ).getObject( 0 ) );
 assertEquals( "not equal: aTuple.get( 1 )", "b", aTuple.get( fields, new Fields( "two" ) ).getObject( 0 ) );
 }

代码示例来源:origin: cwensel/cascading

public void testSetCoerce()
 {
 Fields fieldsA = new Fields( "a", "b", "c" ).applyTypes( String.class, String.class, String.class );
 Tuple tupleA = new Tuple( "0", "1", "2" );
 Fields fieldsB = new Fields( "c", "b" ).applyTypes( Integer.class, Integer.class );
 Tuple tupleB = new Tuple( -2, -1 );
 tupleA.set( fieldsA, fieldsB, tupleB );
 assertEquals( "wrong size", 3, tupleA.size() );
 assertEquals( "not equal: tuple.get(0)", "0", tupleA.getObject( 0 ) );
 assertEquals( "not equal: tuple.get(1)", "-1", tupleA.getObject( 1 ) );
 assertEquals( "not equal: tuple.get(2)", "-2", tupleA.getObject( 2 ) );
 }
}

代码示例来源:origin: cwensel/cascading

@Test
public void testPut()
 {
 Fields aFields = new Fields( "one", "five" );
 tuple.put( fields, aFields, new Tuple( "ten", "eleven" ) );
 assertEquals( "not equal: tuple.size()", 5, tuple.size() );
 assertEquals( "not equal: tuple.get( 0 )", "ten", tuple.getObject( 0 ) );
 assertEquals( "not equal: tuple.get( 0 )", "ten", tuple.get( fields, new Fields( "one" ) ).getObject( 0 ) );
 assertEquals( "not equal: tuple.get( 0 )", "eleven", tuple.getObject( 4 ) );
 assertEquals( "not equal: tuple.get( 0 )", "eleven", tuple.get( fields, new Fields( "five" ) ).getObject( 0 ) );
 }

代码示例来源:origin: cwensel/cascading

private void useSize( TupleEntry input, TupleEntryCollector outputCollector )
 {
 LOG.debug( "using size: {}", size );
 Tuple tuple = new Tuple( input.getTuple() ); // make clone
 Tuple group = tuple.remove( input.getFields(), groupFieldSelector );
 for( int i = 0; i < tuple.size(); i = i + size )
  {
  Tuple result = new Tuple( group );
  result.addAll( tuple.get( Fields.offsetSelector( size, i ).getPos() ) );
  outputCollector.add( result );
  }
 }

代码示例来源:origin: com.hotels/plunger

public Builder<C> addTuple(Object... values) {
 values = FieldTypeValidator.validateValues(fieldMask, values);
 TupleEntry newTuple = new TupleEntry(fields, Tuple.size(fields.size()));
 newTuple.setTuple(fieldMask, new Tuple(values));
 tuples.add(newTuple);
 return this;
}

代码示例来源:origin: cwensel/cascading

/**
 * Method setInteger sets the value in the given field or position.
 *
 * @param fieldName field name or position to set
 * @param value     of type int
 */
public void setInteger( Comparable fieldName, int value )
 {
 int pos = fields.getPos( asFieldName( fieldName ) );
 if( pos > coercions.length - 1 )
  throw new TupleException( "position value is too large: " + pos + ", positions in field: " + tuple.size() );
 tuple.set( pos, coercions[ pos ].canonical( value ) );
 }

代码示例来源:origin: cwensel/cascading

private TupleEntry getEntry( String[] names, Class[] types, Object... values )
 {
 Fields fields = new Fields( names ).applyTypes( types );
 Tuple parameters = new Tuple( values );
 return new TupleEntry( fields, parameters );
 }

代码示例来源:origin: cwensel/cascading

public static Tuple setOnEmpty( TupleEntry baseEntry, TupleEntry valuesEntry )
 {
 Tuple emptyTuple = Tuple.size( baseEntry.getFields().size() );
 emptyTuple.set( baseEntry.getFields(), valuesEntry.getFields(), valuesEntry.getTuple() );
 return emptyTuple;
 }

代码示例来源:origin: cwensel/cascading

@Test
public void testReplace() throws IOException
 {
 RegexReplace splitter = new RegexReplace( new Fields( "words" ), "\\s+", "-", true );
 Tuple arguments = new Tuple( "foo\t bar" );
 Fields resultFields = Fields.UNKNOWN;
 TupleListCollector collector = invokeFunction( splitter, arguments, resultFields );
 assertEquals( "wrong size", 1, collector.size() );
 Iterator<Tuple> iterator = collector.iterator();
 Tuple tuple = iterator.next();
 assertEquals( "not equal: tuple.get(0)", "foo-bar", tuple.getObject( 0 ) );
 }

相关文章