error with storage 28 from storage engine with mysql

Question

I am having an issue with a storage . I recieved this error

 Got error 28 from storage engine 

I have checked the storage capacity and it was still available and it was not full. what can be the reason for this? I have checked everything with no success

It is possible that I am running out of main mysql data directory, or in the mysql tmp. Can someone tell me how to find their place in order to check for it too ?

Answer 1

Once you have found the locations where MySQL stores files, run a command in the shell:

df -Ph <pathname>

Where <pathname> is each of the locations you want to test. Some may be on the same disk volume, so they'll show up as the same when reported by df.

[vagrant@localhost ~]$ mysql -e 'select @@datadir'
+-----------------+
| @@datadir       |
+-----------------+
| /var/lib/mysql/ |
+-----------------+

[vagrant@localhost ~]$ df -Ph /var/lib/mysql
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   38G  3.7G   34G  10% /

This tells me that the disk volume for my datadir is the root volume, including the top-level directory "/" and everything beneath that. The volume is 10% full, with 34G unused.

If the volume where your datadir reaches 100%, then you'll start seeing errno 28 issues when you insert new data and it needs to expand a MySQL tablespace, or write to a log file.

In that case, you need to figure out what's taking so much space. It might be something under the MySQL directory, or like in my case, your datadir might be part of a larger disk volume, where all your other files exist. In that case, any accumulation of files on the system might cause the disk to fill up. For example, log files or temp files even if they're not related to MySQL.

I'd start at the top of the disk volume and use du to figure out which directories are so full.

[vagrant@localhost ~]$ sudo du -shx /*
33M     /boot
34M     /etc
40K     /home
28M     /opt
4.0K    /tmp
2.0G    /usr
941M    /vagrant
666M    /var

Note: if your df command told you that your datadir is on a separate disk volume, you'd start at that volume's mount point. The space used by one disk volume does not count toward another disk volume.

Now I see that /usr is taking the most space, of top-level directories. Drill down and see what's taking space under that:

[vagrant@localhost ~]$ sudo du -shx /usr/*
166M    /usr/bin
126M    /usr/include
345M    /usr/lib
268M    /usr/lib64
55M     /usr/libexec
546M    /usr/local
106M    /usr/sbin
335M    /usr/share
56M     /usr/src

Keep drilling down level by level.

Usually the culprit ends up being pretty clear. Like if you have some huge 500G log file /var/log somewhere that has been growing for months.

An example of a typical culprit is the http server logs.

---

Re your comments:

It sounds like you have a separate storage volume for your database storage. That's good.

You just added the du output to your question above. I see that in your 1.4T disk volume, the largest single file by far is this:

1020G   /vol/db1/mysql/_fool_Gerg_sql_200e_4979_main_16419_2_18.tokudb$

This appears to be a TokuDB tablespace. There's information on how TokuDB handles full disks here: https://www.percona.com/doc/percona-server/LATEST/tokudb/tokudb_faq.html#full-disks

I would not remove those files. I'm not as familiar with TokuDB as I am with InnoDB, but I assume those files are important datafiles. If you remove them, you will lose part of your data and you might corrupt the rest of your data.

I found this post, which explains in detail what the files are used for: https://www.percona.com/blog/2014/07/30/examining-the-tokudb-mysql-storage-engine-file-structure/

The manual also says:

Deleting large numbers of rows from an existing table and then closing the table may free some space, but it may not. Deleting rows may simply leave unused space (available for new inserts) inside TokuDB data files rather than shrink the files (internal fragmentation).

So you can DELETE rows from the table, but the physical file on disk may not shrink. Eventually, you could free enough space that you can build a new TokuDB data file with ALTER TABLE <tablename> ENGINE=TokuDB ROW_FORMAT=TOKUDB_SMALL; (see https://dba.stackexchange.com/questions/48190/tokudb-optimize-table-does-not-reclaim-disk-space)

But this will require enough free disk space to build the new table.

So I'm afraid you have painted yourself into a corner. You no longer have enough disk space to rebuild your large table. You should never let the free disk space get smaller than the space required to rebuild your largest table.

At this point, you probably have to use mysqldump to dump data from your largest table. Not necessarily the whole table, but just what you want to keep (read about the mysqldump --where option). Then DROP TABLE to remove that large table entirely. I assume that will free disk space, where using DELETE won't.

You don't have enough space on your db1 volume to save the dump file, so you'll have to save it to another volume. It looks like you have a larger volume on /vol/cbgb1, but I don't know if it's full.

I'd dump the whole thing, for archiving purposes. Then dump again with a subset.

mkdir /vol/cbgdb1/backup
mysqldump fool Gerg | gzip -c > /vol/cbgdb1/backup/Gerg-dump-full.sql.gz
mysqldump fool Gerg --where "id > 8675309" | gzip -c > /vol/cbgb1/backup/Gerg-dump-partial.sql.gz

I'm totally guessing at the arguments to --where. You'll have to decide how you want to select for a partial dump.

After the big table is dropped, reload the partial data you dumped:

gunzip -c /vol/cbgb1/backup/Gerg-dump-partial.sql.gz | mysql fool

If there are any commands I've given in my examples that you don't already know well, I suggest you learn them before trying them. Or find someone to pair with who is more familiar with those commands.

Answer 2

It is possible that I am running out of main mysql data directory, or in the mysql tmp. Can someone tell me how to find their place in order to check for it too ?

TL;DR

Issue the following commands to inspect the location of your server's data and temporary directories respectively:

SHOW GLOBAL VARIABLES LIKE 'datadir'
SHOW GLOBAL VARIABLES LIKE 'tmpdir'

The values of these variables are typically absolute paths (relative to any chroot jail in which the server is running), but if they happen to be relative paths then they will be relative to the working directory of the process that started the server.

However...

As documented under The MySQL Data Directory (emphasis added):

The following list briefly describes the items typically found in the data directory ...

Some items in the preceding list can be relocated elsewhere by reconfiguring the server. In addition, the --datadir option enables the location of the data directory itself to be changed. For a given MySQL installation, check the server configuration to determine whether items have been moved.

You may therefore also wish to inspect the values of a number of other variables, including:

• pid_file
• ssl_%
• %_log_file
• innodb_data_home_dir
• innodb_log_group_home_dir
• innodb_temp_data_file_path
• innodb_undo_directory
• innodb_buffer_pool_filename

If your server is not responsive...

You can also inspect the server's startup configuration.

As documented under Specifying Program Options, the server's startup configuration is determined "by examining environment variables, then by processing option files, and then by checking the command line" with later options taking precedence over earlier options.

The documentation also lists the locations of the Option Files Read on Unix and Unix-Like Systems, should you require it. Note that the sections of those files that the server reads is determined by the manner in which the server is started, as described in the second and third paragraphs of Server Command Options.