mysql的导入,没有在导入之前去重的设置;
建议方案:
1,全量导入,唯一索引去掉;
2,新增表结构自增id;
3,根据自增id删除重复数据;参考sql如下

删除重复

1
2
3
4
5
6
7
(DELETE a FROM `game_table` a
JOIN (SELECT id FROM game_table GROUP BY TYPE HAVING COUNT(1) > 1) b
WHERE a.id = b.id;)

DELETE a FROM `ce` a
JOIN (SELECT id FROM ce GROUP BY uuid , mesuid HAVING count(1) >1) b
WHERE a.id = b.id;

4,新增唯一索引;

创建全文索引,瞬间使用量超出一倍

1
CREATE FULLTEXT INDEX text_fulltext ON messages(text) with parser ngram;