MySQL 普通表转分区表
版本:
MySQL-5.7.32
前言:
对于业务繁忙的数据库来说，在运行了一定时间后，往往会产生一些数据量较大的表，特别是对于每天新增数据较多的日志表或者流水表，大表对于日常的运维非常的不方便，特别是数据的清理、迁移，表的访问性能也会随着数据量的增大而受到影响，因此，对于大表我们需要进行优化拆分，通常拆分的方案有
所以，通常选择分区表改造方案的主要原因都是为了避免应用层面的改造，对应用层面透明，以及方便日常的运维，前提是表具备改造分区条件。
改造分区前期条件:
参考文档
MySQL官方文档
1 根据业务的场景以及数据的分布，确认是否有匹配的分区表类型以及分区列
- 对于日志表，流水表这种按日期类型进行操作的，可以选择进行range分区
- 对于按用户ID类型的进行操作的，可以选择进行hash，key分区
- 对于按渠道，类型的进行操作的，可以选择进行list分区
2 应用涉及的sql，需要90%以上的操作都包含分区列，按分区操作，如果sql没有包含分区条件，扫描全分区，性能会出现下降。
查询表sql操作历史
select db,query,exec_count from sys.x$statement_analysis where lower(query) like '%%'order by exec_count;3 主键必须包含分区键
4 分区键表达式只支持部分函数，存储过程，不支持二级制操作符以及/
5 分区不支持外键
查询表外键
select * from information_schema.KEY_COLUMN_USAGEwhere constraint_schema ='' and REFERENCED_TABLE_SCHEMA is not null\G6 不支持查询缓存
7 5.7版本单个表分区最大支持8192个，并且会话第一次访问分区表，都需要打开全部的分区表，所以避免建立过多的分区
8 数据库最大文件打开数open_files_limit要设置足够大以满足表，分区的打开数量
9 数据库大文件large_files_support设置为on
10 分区列支持null值(对于rang分区，null值总小于任何的一个非null值，即存放在最左边的分区;对于list分区需要显示指定null值条件)，但从数据管理以及规范来看，不建议分区列存放null值，并且如果表有主键，则分区列不能为null值，因为分区列需要作为主键的一部分，不能为null
12 MySQL目前没有自动分区功能，所以需要监控分区的使用情况，通过job自动或者定时手动添加新分区
13 确定数据保留期限，定期归档分区数据
分区改造案例:
以下是一张数据量为766万的大表xxxx_user. xxxx_tab,计划将其改造为范围分区，按月存放。
select table_schema,table_name,table_rows,data_length/1024/1024/1024from information_schema.tableswhere table_name='xxxx_tab';
表结构
CREATE TABLE `xxxx_tab` ( `ROLE_SEQ` bigint(20) NOT NULL , `PRD_ID` varchar(64) NOT NULL , `MAKE_RIGHT` varchar(1) DEFAULT '0' , `CHECK_RIGHT` varchar(1) DEFAULT '0' , `AUTH_RIGHT` varchar(1) DEFAULT '0' , `AUTH_GROUP` varchar(4) DEFAULT NULL , `RELEASE_RIGHT` varchar(1) DEFAULT '0' , `CREATE_USER_SEQ` bigint(20) DEFAULT NULL , `CREATE_DEPT_SEQ` bigint(20) DEFAULT NULL , `CREATE_TIME` datetime DEFAULT NULL , `UPDATE_USER_SEQ` bigint(20) DEFAULT NULL , `UPDATE_DEPT_SEQ` bigint(20) DEFAULT NULL , `UPDATE_TIME` datetime DEFAULT NULL , PRIMARY KEY (`PRD_ID`,`ROLE_SEQ`), KEY `xxxx_tab_IDX01` (`ROLE_SEQ`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4分区列create_time日期最大值，最小值，根据这个范围按月创建分区
select max(CREATE_TIME),min(CREATE_TIME)from xxxx_tab;
分区列null值，对于存在的null值，需要应用对null数据进行处理，并且程序上需要确保数据写入not null
select count(*)from xxxx_tabwhere create_time is null; 
主键重建添加分区列
alter table xxxx_tab drop primary key,add primary key (`PRD_ID`,`ROLE_SEQ`,`CREATE_TIME`);由于主键没有包含分区列，需要重建主键添加分区列，对于主键重建我采取的是官方的online ddl功能，这种ddl操作会造成主从延时，但是不会产生大量的binlog，对于主从实时性要求高的，可以采用第三方的在线工具pt-osc,gh-ost
表转化为分区表
采用pt-osc在线将表转化为分区表，对于partition by 官方是不支持online ddl的，所以需要采用第三方的在线工具
./pt-online-schema-change --user=xxx --password=xxx --charset=utf8 D=xxxx_user,t=xxxx_tab --alter "PARTITION BY RANGE COLUMNS(CREATE_TIME)(PARTITION p200001 VALUES LESS THAN ('2000-02-01 00:00:00') ENGINE = InnoDB, PARTITION p200101 VALUES LESS THAN ('2001-02-01 00:00:00') ENGINE = InnoDB, PARTITION p201707 VALUES LESS THAN ('2017-08-01 00:00:00') ENGINE = InnoDB, PARTITION p201708 VALUES LESS THAN ('2017-09-01 00:00:00') ENGINE = InnoDB, PARTITION p201709 VALUES LESS THAN ('2017-10-01 00:00:00') ENGINE = InnoDB, PARTITION p201710 VALUES LESS THAN ('2017-11-01 00:00:00') ENGINE = InnoDB, PARTITION p201711 VALUES LESS THAN ('2017-12-01 00:00:00') ENGINE = InnoDB, PARTITION p201712 VALUES LESS THAN ('2018-01-01 00:00:00') ENGINE = InnoDB, PARTITION p201801 VALUES LESS THAN ('2018-02-01 00:00:00') ENGINE = InnoDB, PARTITION p201802 VALUES LESS THAN ('2018-03-01 00:00:00') ENGINE = InnoDB, PARTITION p201803 VALUES LESS THAN ('2018-04-01 00:00:00') ENGINE = InnoDB, PARTITION p201804 VALUES LESS THAN ('2018-05-01 00:00:00') ENGINE = InnoDB, PARTITION p201805 VALUES LESS THAN ('2018-06-01 00:00:00') ENGINE = InnoDB, PARTITION p201806 VALUES LESS THAN ('2018-07-01 00:00:00') ENGINE = InnoDB, PARTITION p201807 VALUES LESS THAN ('2018-08-01 00:00:00') ENGINE = InnoDB, PARTITION p201808 VALUES LESS THAN ('2018-09-01 00:00:00') ENGINE = InnoDB, PARTITION p201809 VALUES LESS THAN ('2018-10-01 00:00:00') ENGINE = InnoDB, PARTITION p201810 VALUES LESS THAN ('2018-11-01 00:00:00') ENGINE = InnoDB, PARTITION p201811 VALUES LESS THAN ('2018-12-01 00:00:00') ENGINE = InnoDB, PARTITION p201812 VALUES LESS THAN ('2019-01-01 00:00:00') ENGINE = InnoDB, PARTITION p201901 VALUES LESS THAN ('2019-02-01 00:00:00') ENGINE = InnoDB, PARTITION p201902 VALUES LESS THAN ('2019-03-01 00:00:00') ENGINE = InnoDB, PARTITION p201903 VALUES LESS THAN ('2019-04-01 00:00:00') ENGINE = InnoDB, PARTITION p201904 VALUES LESS THAN ('2019-05-01 00:00:00') ENGINE = InnoDB, PARTITION p201905 VALUES LESS THAN ('2019-06-01 00:00:00') ENGINE = InnoDB, PARTITION p201906 VALUES LESS THAN ('2019-07-01 00:00:00') ENGINE = InnoDB, PARTITION p201907 VALUES LESS THAN ('2019-08-01 00:00:00') ENGINE = InnoDB, PARTITION p201908 VALUES LESS THAN ('2019-09-01 00:00:00') ENGINE = InnoDB, PARTITION p201909 VALUES LESS THAN ('2019-10-01 00:00:00') ENGINE = InnoDB, PARTITION p201910 VALUES LESS THAN ('2019-11-01 00:00:00') ENGINE = InnoDB, PARTITION p201911 VALUES LESS THAN ('2019-12-01 00:00:00') ENGINE = InnoDB, PARTITION p201912 VALUES LESS THAN ('2020-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202001 VALUES LESS THAN ('2020-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202002 VALUES LESS THAN ('2020-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202003 VALUES LESS THAN ('2020-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202004 VALUES LESS THAN ('2020-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202005 VALUES LESS THAN ('2020-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202006 VALUES LESS THAN ('2020-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202007 VALUES LESS THAN ('2020-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202008 VALUES LESS THAN ('2020-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202009 VALUES LESS THAN ('2020-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202010 VALUES LESS THAN ('2020-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202011 VALUES LESS THAN ('2020-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202012 VALUES LESS THAN ('2021-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202101 VALUES LESS THAN ('2021-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202102 VALUES LESS THAN ('2021-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202103 VALUES LESS THAN ('2021-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202104 VALUES LESS THAN ('2021-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202105 VALUES LESS THAN ('2021-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202106 VALUES LESS THAN ('2021-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202107 VALUES LESS THAN ('2021-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202108 VALUES LESS THAN ('2021-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202109 VALUES LESS THAN ('2021-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202110 VALUES LESS THAN ('2021-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202111 VALUES LESS THAN ('2021-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202112 VALUES LESS THAN ('2022-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202201 VALUES LESS THAN ('2022-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202202 VALUES LESS THAN ('2022-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202203 VALUES LESS THAN ('2022-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202204 VALUES LESS THAN ('2022-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202205 VALUES LESS THAN ('2022-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202206 VALUES LESS THAN ('2022-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202207 VALUES LESS THAN ('2022-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202208 VALUES LESS THAN ('2022-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202209 VALUES LESS THAN ('2022-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202210 VALUES LESS THAN ('2022-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202211 VALUES LESS THAN ('2022-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202212 VALUES LESS THAN ('2023-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202301 VALUES LESS THAN ('2023-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202302 VALUES LESS THAN ('2023-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202303 VALUES LESS THAN ('2023-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202304 VALUES LESS THAN ('2023-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202305 VALUES LESS THAN ('2023-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202306 VALUES LESS THAN ('2023-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202307 VALUES LESS THAN ('2023-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202308 VALUES LESS THAN ('2023-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202309 VALUES LESS THAN ('2023-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202310 VALUES LESS THAN ('2023-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202311 VALUES LESS THAN ('2023-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202312 VALUES LESS THAN ('2024-01-01 00:00:00') ENGINE = InnoDB, PARTITION pmax VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB)" --recursion-method hosts --max-lag 600 --nodrop-old-table --print --statistics --execute分区后表模型
CREATE TABLE `xxxx_tab` ( `ROLE_SEQ` bigint(20) NOT NULL , `PRD_ID` varchar(64) NOT NULL , `MAKE_RIGHT` varchar(1) DEFAULT '0' , `CHECK_RIGHT` varchar(1) DEFAULT '0' , `AUTH_RIGHT` varchar(1) DEFAULT '0' , `AUTH_GROUP` varchar(4) DEFAULT NULL , `RELEASE_RIGHT` varchar(1) DEFAULT '0' , `CREATE_USER_SEQ` bigint(20) DEFAULT NULL , `CREATE_DEPT_SEQ` bigint(20) DEFAULT NULL , `CREATE_TIME` datetime NOT NULL , `UPDATE_USER_SEQ` bigint(20) DEFAULT NULL , `UPDATE_DEPT_SEQ` bigint(20) DEFAULT NULL , `UPDATE_TIME` datetime DEFAULT NULL , PRIMARY KEY (`PRD_ID`,`ROLE_SEQ`,`CREATE_TIME`), KEY `xxxx_tab_IDX01` (`ROLE_SEQ`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 PARTITION BY RANGE COLUMNS(CREATE_TIME)(PARTITION p200001 VALUES LESS THAN ('2000-02-01 00:00:00') ENGINE = InnoDB, PARTITION p200101 VALUES LESS THAN ('2001-02-01 00:00:00') ENGINE = InnoDB, PARTITION p201707 VALUES LESS THAN ('2017-08-01 00:00:00') ENGINE = InnoDB, PARTITION p201708 VALUES LESS THAN ('2017-09-01 00:00:00') ENGINE = InnoDB, PARTITION p201709 VALUES LESS THAN ('2017-10-01 00:00:00') ENGINE = InnoDB, PARTITION p201710 VALUES LESS THAN ('2017-11-01 00:00:00') ENGINE = InnoDB, PARTITION p201711 VALUES LESS THAN ('2017-12-01 00:00:00') ENGINE = InnoDB, PARTITION p201712 VALUES LESS THAN ('2018-01-01 00:00:00') ENGINE = InnoDB, PARTITION p201801 VALUES LESS THAN ('2018-02-01 00:00:00') ENGINE = InnoDB, PARTITION p201802 VALUES LESS THAN ('2018-03-01 00:00:00') ENGINE = InnoDB, PARTITION p201803 VALUES LESS THAN ('2018-04-01 00:00:00') ENGINE = InnoDB, PARTITION p201804 VALUES LESS THAN ('2018-05-01 00:00:00') ENGINE = InnoDB, PARTITION p201805 VALUES LESS THAN ('2018-06-01 00:00:00') ENGINE = InnoDB, PARTITION p201806 VALUES LESS THAN ('2018-07-01 00:00:00') ENGINE = InnoDB, PARTITION p201807 VALUES LESS THAN ('2018-08-01 00:00:00') ENGINE = InnoDB, PARTITION p201808 VALUES LESS THAN ('2018-09-01 00:00:00') ENGINE = InnoDB, PARTITION p201809 VALUES LESS THAN ('2018-10-01 00:00:00') ENGINE = InnoDB, PARTITION p201810 VALUES LESS THAN ('2018-11-01 00:00:00') ENGINE = InnoDB, PARTITION p201811 VALUES LESS THAN ('2018-12-01 00:00:00') ENGINE = InnoDB, PARTITION p201812 VALUES LESS THAN ('2019-01-01 00:00:00') ENGINE = InnoDB, PARTITION p201901 VALUES LESS THAN ('2019-02-01 00:00:00') ENGINE = InnoDB, PARTITION p201902 VALUES LESS THAN ('2019-03-01 00:00:00') ENGINE = InnoDB, PARTITION p201903 VALUES LESS THAN ('2019-04-01 00:00:00') ENGINE = InnoDB, PARTITION p201904 VALUES LESS THAN ('2019-05-01 00:00:00') ENGINE = InnoDB, PARTITION p201905 VALUES LESS THAN ('2019-06-01 00:00:00') ENGINE = InnoDB, PARTITION p201906 VALUES LESS THAN ('2019-07-01 00:00:00') ENGINE = InnoDB, PARTITION p201907 VALUES LESS THAN ('2019-08-01 00:00:00') ENGINE = InnoDB, PARTITION p201908 VALUES LESS THAN ('2019-09-01 00:00:00') ENGINE = InnoDB, PARTITION p201909 VALUES LESS THAN ('2019-10-01 00:00:00') ENGINE = InnoDB, PARTITION p201910 VALUES LESS THAN ('2019-11-01 00:00:00') ENGINE = InnoDB, PARTITION p201911 VALUES LESS THAN ('2019-12-01 00:00:00') ENGINE = InnoDB, PARTITION p201912 VALUES LESS THAN ('2020-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202001 VALUES LESS THAN ('2020-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202002 VALUES LESS THAN ('2020-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202003 VALUES LESS THAN ('2020-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202004 VALUES LESS THAN ('2020-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202005 VALUES LESS THAN ('2020-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202006 VALUES LESS THAN ('2020-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202007 VALUES LESS THAN ('2020-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202008 VALUES LESS THAN ('2020-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202009 VALUES LESS THAN ('2020-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202010 VALUES LESS THAN ('2020-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202011 VALUES LESS THAN ('2020-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202012 VALUES LESS THAN ('2021-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202101 VALUES LESS THAN ('2021-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202102 VALUES LESS THAN ('2021-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202103 VALUES LESS THAN ('2021-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202104 VALUES LESS THAN ('2021-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202105 VALUES LESS THAN ('2021-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202106 VALUES LESS THAN ('2021-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202107 VALUES LESS THAN ('2021-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202108 VALUES LESS THAN ('2021-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202109 VALUES LESS THAN ('2021-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202110 VALUES LESS THAN ('2021-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202111 VALUES LESS THAN ('2021-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202112 VALUES LESS THAN ('2022-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202201 VALUES LESS THAN ('2022-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202202 VALUES LESS THAN ('2022-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202203 VALUES LESS THAN ('2022-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202204 VALUES LESS THAN ('2022-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202205 VALUES LESS THAN ('2022-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202206 VALUES LESS THAN ('2022-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202207 VALUES LESS THAN ('2022-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202208 VALUES LESS THAN ('2022-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202209 VALUES LESS THAN ('2022-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202210 VALUES LESS THAN ('2022-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202211 VALUES LESS THAN ('2022-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202212 VALUES LESS THAN ('2023-01-01 00:00:00') ENGINE = InnoDB, PARTITION p202301 VALUES LESS THAN ('2023-02-01 00:00:00') ENGINE = InnoDB, PARTITION p202302 VALUES LESS THAN ('2023-03-01 00:00:00') ENGINE = InnoDB, PARTITION p202303 VALUES LESS THAN ('2023-04-01 00:00:00') ENGINE = InnoDB, PARTITION p202304 VALUES LESS THAN ('2023-05-01 00:00:00') ENGINE = InnoDB, PARTITION p202305 VALUES LESS THAN ('2023-06-01 00:00:00') ENGINE = InnoDB, PARTITION p202306 VALUES LESS THAN ('2023-07-01 00:00:00') ENGINE = InnoDB, PARTITION p202307 VALUES LESS THAN ('2023-08-01 00:00:00') ENGINE = InnoDB, PARTITION p202308 VALUES LESS THAN ('2023-09-01 00:00:00') ENGINE = InnoDB, PARTITION p202309 VALUES LESS THAN ('2023-10-01 00:00:00') ENGINE = InnoDB, PARTITION p202310 VALUES LESS THAN ('2023-11-01 00:00:00') ENGINE = InnoDB, PARTITION p202311 VALUES LESS THAN ('2023-12-01 00:00:00') ENGINE = InnoDB, PARTITION p202312 VALUES LESS THAN ('2024-01-01 00:00:00') ENGINE = InnoDB, PARTITION pmax VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB)