nodejs mysql优化查询

4dbbbstv  于 2021-06-21  发布在  Mysql
关注(0)|答案(2)|浏览(316)

我正在使用 mysql2 NodeJS v8.9.4中的模块。
此函数用于从满足以下条件的消息队列中获取消息: status==0 如果计数 botIdstatus==1 小于10
如果 retry_afterwaitbotId+chatId 只是 botId 小于 NOW (时间戳)
如果没有相同的 chatIdstatus==1 ```
static async Find(activeMessageIds, maxActiveMsgPerBot) {

let params                 = [maxActiveMsgPerBot];
let filterActiveMessageIds = ' ';
let time                   = Util.GetTimeStamp();

if (activeMessageIds && activeMessageIds.length) {
  filterActiveMessageIds = 'q.id NOT IN (?) AND ';
  params.push(activeMessageIds);
}

let q =
      `select q.* 
        from bot_message_queue q 
        left join bot_message_queue_wait w on q.botId=w.botId AND q.chatId=w.chatId 
        left join bot_message_queue_wait w2 on q.botId=w2.botId AND w2.chatId=0 
        where
          q.status=0 AND
          q.botId NOT IN (select q2.botId from bot_message_queue q2 where q2.status=1 group by q2.botId HAVING COUNT(q2.botId)>?) AND 
          ${filterActiveMessageIds} 
          q.chatId NOT IN (select q3.chatId from bot_message_queue q3 where q3.status=1 group by q3.chatId) AND 
          (w.retry_after IS NULL OR w.retry_after <= ?) AND 
          (w2.retry_after IS NULL OR w2.retry_after <= ?)  
        order by q.priority DESC,q.id ASC  
        limit 1;`;

params.push(time);
params.push(time);

let con    = await DB.connection();
let result = await DB.query(q, params, con);
if (result && result.length) {
  result      = result[0];
  let updateQ = `update bot_message_queue set status=1 where id=?;`;
  await DB.query(updateQ, [result.id], con);
} else
  result = null;

con.release();
return result;

}

这个查询在我的本地开发系统上运行良好。它在服务器phpmyadmin中也可以在几毫秒内正常运行。
但是当它运行throw nodejs+mysql2时,cpu使用率会上升到100%,这个表中只有2k行。

CREATE TABLE IF NOT EXISTS bot_message_queue (
id int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
botId int(10) UNSIGNED NOT NULL,
chatId varchar(50) CHARACTER SET utf8 NOT NULL,
type varchar(50) DEFAULT NULL,
message longtext NOT NULL,
add_date int(10) UNSIGNED NOT NULL,
status tinyint(2) UNSIGNED NOT NULL DEFAULT '0' COMMENT '0=waiting,1=sendig,2=sent,3=error',
priority tinyint(1) UNSIGNED NOT NULL DEFAULT '5' COMMENT '5=normal messages,<5 = bulk messages',
delay_after int(10) UNSIGNED NOT NULL DEFAULT '1000',
send_date int(10) UNSIGNED DEFAULT NULL,
PRIMARY KEY (id),
KEY botId (botId,status),
KEY botId_2 (botId,chatId,status,priority),
KEY chatId (chatId,status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

CREATE TABLE IF NOT EXISTS bot_message_queue_wait (
botId int(10) UNSIGNED NOT NULL,
chatId varchar(50) CHARACTER SET utf8 NOT NULL,
retry_after int(10) UNSIGNED NOT NULL,
PRIMARY KEY (botId,chatId),
KEY retry_after (retry_after),
KEY botId (botId,chatId,retry_after)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

更新:此处的实际表数据更新2:
获取消息时间:
-最小值:1788 ms
-最大:44285毫秒
-平均值:20185.4 ms
到昨天为止,最大值是20秒:(现在是40秒!!!
更新3:我合并了这两个连接,其中:

left join bot_message_queue_wait w on q.botId=w.botId AND q.chatId=w.chatId
left join bot_message_queue_wait w2 on q.botId=w2.botId AND w2.chatId=0

(w.retry_after IS NULL OR w.retry_after <= ?) AND
(w2.retry_after IS NULL OR w2.retry_after <= ?)

变成一个单一的,我希望这将工作的预期!

left join bot_message_queue_wait w on q.botId=w.botId AND ( q.chatId=w.chatId OR w.chatId=0 )

暂时我去掉了2个where,查询时间恢复正常。

q.botId NOT IN (select ...)
q.chatId NOT IN (select ...)

所以这两个查询是阻塞点,需要修复。
rdlzhqv9

rdlzhqv91#

在本例中,我将用not exists替换not in子查询,因为它可以执行得更好。
将顺序切换为all desc或all asc
因此,要优化查询,首先要添加以下索引:

ALTER TABLE `bot_message_queue` ADD INDEX `bot_message_queue_idx_status_botid_chatid_priori_id` (`status`,`botId`,`chatId`,`priority`,`id`);
ALTER TABLE `bot_message_queue` ADD INDEX `bot_message_queue_idx_priority_id` (`priority`,`id`);
ALTER TABLE `bot_message_queue` ADD INDEX `bot_message_queue_idx_botid_status` (`botId`,`status`);
ALTER TABLE `bot_message_queue` ADD INDEX `bot_message_queue_idx_chatid_status` (`chatId`,`status`);
ALTER TABLE `bot_message_queue_wait` ADD INDEX `bot_message_queue_wa_idx_chatid_botid` (`chatId`,`botId`);

现在,您可以尝试运行此查询(请注意,我将order by更改为all desc,因此如果需要,您可以将其更改为asc):

SELECT
        bot_message_queue.* 
    FROM
        bot_message_queue q 
    LEFT JOIN
        bot_message_queue_wait w 
            ON q.botId = w.botId 
            AND q.chatId = w.chatId 
    LEFT JOIN
        bot_message_queue_wait w2 
            ON q.botId = w2.botId 
            AND w2.chatId = 0 
    WHERE
        q.status = 0 
        AND NOT EXISTS (
            SELECT
                1 
            FROM
                bot_message_queue AS q21 
            WHERE
                q21.status = 1 
                AND q.botId = q21.botId 
            GROUP BY
                q21.botId 
            HAVING
                COUNT(q21.botId) > ? 
            ORDER BY
                NULL
        ) 
        AND NOT EXISTS (
            SELECT
                1 
            FROM
                bot_message_queue AS q32 
            WHERE
                q32.status = 1 
                AND q.chatId = q32.chatId 
            GROUP BY
                q32.chatId 
            ORDER BY
                NULL
        ) 
        AND (
            w.retry_after IS NULL 
            OR w.retry_after <= ?
        ) 
        AND (
            w2.retry_after IS NULL 
            OR w2.retry_after <= ?
        ) 
    ORDER BY
        q.priority DESC,
        q.id DESC LIMIT 1
snvhrwxg

snvhrwxg2#

NOT IN ( SELECT ... ) 很难优化。 OR 无法优化。
ORDER BY ,混合 DESC 以及 ASC 不再使用索引(直到8.0)。考虑改变 ASCDESC . 之后, INDEX(priority, id) 可能会有帮助。
是什么 ${filterActiveMessageIds} ?
这个 GROUP BY 中不需要

NOT IN ( SELECT  q3.chatId
            from  bot_message_queue q3
            where  q3.status=1
            group by  q3.chatId )
``` `INDEX(status, chatid)` 按此顺序将有利于该子查询。 `INDEX(status, botid)` 按此顺序
有关创建索引的详细信息:http://mysql.rjweb.org/doc.php/index_cookbook_mysql

相关问题