CODEWITHSUNDEEP
mysqlpartitiondatabase-partitioning
Edosam
Edosam

Reputation: 11

Partiton mysql table by column timestamp

I am trying to partition my table MySQL innoDB. Right now there are approximately 2 million rows in the location table (and growing always) rows of history data. I must perodicly delete the dataset old by year I use MySQL 5.7.22 Community Server.

CREATE TABLE `geo_data` (
  `ID` bigint(20) NOT NULL AUTO_INCREMENT,
  `ID_DISP` bigint(20) DEFAULT NULL,
  `SYS_TIMESTAMP` datetime DEFAULT NULL,
  `DATA_TIMESTAMP` bigint(20) DEFAULT NULL,
  `X` double DEFAULT NULL,
  `Y` double DEFAULT NULL,
  `SPEED` bigint(20) DEFAULT NULL,
  `HEADING` bigint(20) DEFAULT NULL,
  `ID_DATA_TYPE` bigint(20) DEFAULT NULL,
  `PROCESSED` bigint(20) DEFAULT NULL,
  `ALTITUDE` bigint(20) DEFAULT NULL,
  `ID_UNIT` bigint(20) DEFAULT NULL,
  `ID_DRIVER` bigint(20) DEFAULT NULL,
  UNIQUE KEY `part_id` (`ID`,`DATA_TIMESTAMP`,`ID_DISP`),
  KEY `Index_idDisp_dataTS_type` (`ID_DISP`,`DATA_TIMESTAMP`,`ID_DATA_TYPE`),
  KEY `Index_idDisp_dataTS` (`ID_DISP`,`DATA_TIMESTAMP`),
  KEY `Index_TS` (`DATA_TIMESTAMP`),
  KEY `idx_sysTS_idDisp` (`ID_DISP`,`SYS_TIMESTAMP`),
  KEY `idx_clab_geo_data_ID_UNIT_DATA_TIMESTAMP_ID_DATA_TYPE` (`ID_UNIT`,`DATA_TIMESTAMP`,`ID_DATA_TYPE`),
  KEY `idx_idUnit_dataTS` (`ID_UNIT`,`DATA_TIMESTAMP`),
  KEY `idx_clab_geo_data_ID_DRIVER_DATA_TIMESTAMP_ID_DATA_TYPE` (`ID_DRIVER`,`DATA_TIMESTAMP`,`ID_DATA_TYPE`)
) ENGINE=InnoDB AUTO_INCREMENT=584390 DEFAULT CHARSET=latin1;

I have to partition by DATA_TIMESTAMP (format timestamp date gps).

ALTER TABLE geo_data
PARTITION BY RANGE (year(from_unixtime(data_timestamp)))
(
   PARTITION p2018 VALUES LESS THAN ('2018'),
   PARTITION p2019 VALUES LESS THAN ('2019'),
   PARTITION pmax VALUES LESS THAN MAXVALUE
);

Error Code: 1697. VALUES value for partition 'p2018' must have type INT

How can I do?

I would like to add later a subpartion range by ID_DISP. How can I do?

Thanks in advance!

Upvotes: 1

Views: 1980

Answers (2)

Rick James
Rick James

Reputation: 142296

Since data_timestamp was actually a BIGINT, you are not permitted to use date functions. It seemed there were two errors, and this probably fixes them:

ALTER TABLE geo_data
PARTITION BY RANGE (data_timestamp)
(
   PARTITION p2018 VALUES LESS THAN (UNIX_TIMESTAMP('2018-01-01') * 1000),
   PARTITION p2019 VALUES LESS THAN (UNIX_TIMESTAMP('2019-01-01') * 1000),
   PARTITION pmax VALUES LESS THAN MAXVALUE
);

I am assuming your data_timestamp is really milliseconds, a la Java? If not, the decide what to do with the * 1000.

SUBPARTITIONs are useless; don't bother with them. If you really want to partition by Month or Quarter, then simply do it at the PARTITION level.

Recommendation: Don't have more than about 50 partitions.

How many "drivers" do you have? I suspect you do not have trillions. So, don't blindly use BIGINT for ids. Each takes 8 bytes. SMALLINT UNSIGNED, for example, would take only 2 bytes and allow 64K drivers (etc).

If X and Y are latitude and longitude, it might be clearer to name them such. Here are what datatype to use instead of the 8-byte DOUBLE, depending on the resolution you have (and need). 4-byte FLOATs are probably good enough for vehicles.

The table has several redundant indexes; toss them. Also, note that when you have INDEX(a,b,c), it is redundant to also have INDEX(a,b).

See also my discussion on Partitioning, especially related to time-series, such as yours.

Hmmm... I wonder if the 63 bits of precision for SPEED will let you record them when they go at the speed of light?

Another point: Don't create p2019 until just before the start of 2019. You have pmax in case you goof and fail to add that partition in time. And the REORGANIZE PARTITION mentioned in my discussion covers how to recover from such a goof.

Upvotes: 2

Manse
Manse

Reputation: 38147

Update:

Seems you cannot use from_unixtime in a PARTITION BY RANGE query because hash partitions must be based on an integer expression. More info see this answer

Its expecting an INT not a STRING (as per the error message), so try :

ALTER TABLE geo_data
PARTITION BY RANGE (year(from_unixtime(data_timestamp)))
(
   PARTITION p2018 VALUES LESS THAN (2018),
   PARTITION p2019 VALUES LESS THAN (2019),
   PARTITION pmax VALUES LESS THAN MAXVALUE
);

Here I have specified the year in the partition values as an int ie 2018 / 2019, and not strings as in '2018' / '2019'

Upvotes: 0

Related Questions