incubator-doris Spill() is disabled, but can be called

sr4lhrrt  于 2022-04-22  发布在  Java
关注(0)|答案(0)|浏览(197)

Describe the bug

We got an error:
ERROR 1064 (HY000): Memory exceed limit. New partitioned Aggregation in spill Backend: xxx.xxx.xxx.xxx, fragment: xxx Used: 10528896, Limit: -1. You can change the limit by session variable exec_mem_limit.

incubator-doris/be/src/exec/new_partitioned_aggregation_node.cc

Lines 816 to 824 in 14c0881

| | Status NewPartitionedAggregationNode::Partition::Spill(bool more_aggregate_rows) { |
| | DCHECK(!parent->is_streaming_preagg_); |
| | DCHECK(!is_closed); |
| | DCHECK(!is_spilled()); |
| | // TODO(ml): enable spill |
| | std::stringstream msg; |
| | msg << "New partitioned Aggregation in spill"; |
| | LIMIT_EXCEEDED(parent->mem_tracker(), parent->state_, msg.str()); |
| | // RETURN_IF_ERROR(parent->state_->StartSpilling(parent->mem_tracker())); |

may caused by
incubator-doris/be/src/exec/new_partitioned_aggregation_node.cc

Lines 1131 to 1138 in 14c0881

| | bool got_memory; |
| | RETURN_IF_ERROR(partition->InitHashTable(&got_memory)); |
| | // Spill the partition if we cannot create a hash table for a merge aggregation. |
| | if (UNLIKELY(!got_memory)) { |
| | DCHECK(!is_streaming_preagg_) << "Preagg reserves enough memory for hash tables"; |
| | // If we're repartitioning, we will be writing aggregated rows first. |
| | RETURN_IF_ERROR(partition->Spill(level > 0)); |
| | } |

But whatever caused Spill() calling, if Spill() is disabled(), we shouldn't call it, right? And the error message is misleading.

Solutions

  1. Spill() return the right error msg "Spill is disabled. And the reason maybe mem limit... ". LIMIT_EXCEEDED just get one mem_tracker's limit, it's incorrect, we need to use lowest_limit().
    PS. If Spill() is called coz Suballoctor->allocate() failed(got_memory==false), it may be not related to this plan's mem_trackers limit.
  2. enable Spill(), it's unfamiliar to me, so we need more thorough discuss.

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题