incubator-doris Support loading data into mutil tables in one load execution plan

enxuqcxy  于 2022-04-22  发布在  Java
关注(0)|答案(0)|浏览(110)

Currently, Doris only supports loading data into one table in a load execution plan.
Which means if user want to load the data from a file to, for example, 4 tables, it has to
generate 4 load execution plans, and each plan will read the data from file alone.

It is inefficient. So I would like to implement a new load execution plan which contains 2
new plan nodes:

  1. DataDuplicateNode
    DataDuplicateNode will get data from its child (such as BrokerScanNode), and duplicate
    these data for target tuples.
  2. MultiOlapTableSink
    MultiOlapTableSink is a DataSink, which contains several OlapTableSink.
    It should be the sink of DataDuplicateNode.

The plan fragment should looks like:

+--------------------------+
|PlanFragment              |
|                          |
|    +-------------------+ |
|    |MultiOlapTableSink | |
|    |                   | |
|    |   OlapTableSink +-----> TabletsChannel
|    |                   | |
|    |   OlapTableSink +-----> TabletsChannel
|    |                   | |
|    |   OlapTableSink +-----> TabletsChannel
|    +---------^---------+ |
|              |           |
|              | 2. Send   |
|              |           |
|     +--------+--------+  |
|     |                 |  |
|     |DataDuplicateNode|  |
|     |                 |  |
|     +--------+--------+  |
|              |           |
|              | 1. GetNext|
|              |           |
|     +--------v--------+  |
|     |                 |  |
|     |  BrokerScanNode |  |
|     |                 |  |
|     +-----------------+  |
+--------------------------+

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题