In the latest MySQL 5.6 Labs release we have implemented an InnoDB buffer pool(s) dump and load to solve this problem.

The contents of the InnoDB buffer pool(s) can be saved on disk before MySQL is shut down and then read in after a restart so that the warm up time is drastically shortened – the buffer pool(s) go to the state they were before the server restart! The time needed for that is roughly the time needed to read data from disk that is about the size of the buffer pool(s).

Lets dive straight into the commands to perform various dump/load operations:

The buffer pool(s) dump can be done at any time when MySQL is running by doing:

  mysql> SET innodb_buffer_pool_dump_now=ON;

This operation is very quick and creates a file named ib_buffer_pool in the InnoDB data directory, by analogy with the default InnoDB data file ibdata1. The file contains only the space and page IDs and is about 500k per gigabyte of buffer pool. The same operation can also be done automatically during MySQL shutdown:

  mysql> SET innodb_buffer_pool_dump_at_shutdown=ON;

By analogy the buffer pool(s) can be loaded manually by:

  mysql> SET innodb_buffer_pool_load_now=ON;

and automatically during startup from the file ib_buffer_pool by setting

  innodb_buffer_pool_load_at_startup=ON

in my.cnf or my.ini.

The filename where the buffer pool(s) are dumped and later loaded from can be specified by changing the value of the global configuration variable innodb_buffer_pool_filename which defaults to ib_buffer_pool.

Additionally the progress of the buffer pool(s) dump (although very quick) and load (not so quick) can be monitored by:

  mysql> SHOW STATUS LIKE 'innodb_buffer_pool_dump_status';

  mysql> SHOW STATUS LIKE 'innodb_buffer_pool_load_status';

And finally the load operation can be interrupted by:

  mysql> SET innodb_buffer_pool_load_abort=ON;

Before loading the list is sorted so that pages will be read in as close to sequential order as possible. The load is asynchronous so it should not interfere with normal operations too much. The load is clever enough to handle different buffer pool sizes or pages that are missing during the load. A page will not be loaded if it is already in the buffer pool so there is no need to worry about problems with changed pages.

The dump file is plain text so it is possible to edit it, though we do not recommend this.

Enjoy!

InnoDB gathers statistics for the data in user tables, which are used by the MySQL optimizer to choose the best query plan. For a long time the imprecision and instability of these statistics have been creating problems for users.

The problem is that these statistics are recalculated at any of the following events:

* When the table is opened

* When the table has changed a lot (1/16th of the table has been updated/deleted or inserted)

* When ANALYZE TABLE is run

* When SHOW TABLE STATUS or SHOW INDEX FROM is run

* When InnoDB Monitor is turned ON

* others

and so their recalculation must be quick an unnoticeable. Thus the quick algorithm just picks 8 random pages and could give a wildly varying results.

Quick glance

To solve this problem MySQL 5.6.2 introduces the InnoDB Persistent Statistics feature, which:

* Uses a more precise sampling algorithm (which is also slower) in an attempt to better inform the MySQL optimizer so it can choose the best possible query plan.

* Does not recalculate the statistics automatically, only when ANALYZE TABLE is run. This means that the same query plan will always be chosen even when MySQL is restarted (see below). A query plan using the global statistics can only change after ANALYZE TABLE has been run (manually). This also means that the user and/or DBA is responsible for running ANALYZE TABLE regularly if the data in the table has been changed a lot.

* Saves the statistics on disk in normal user-visible and updatable SQL tables.

How to turn it ON?

To turn ON the persistent statistics for a given table you must first create the persistent statistics storage by executing the SQL script storage/innobase/scripts/persistent_storage.sql which is shipped in the source distribution (in future versions it may be executed automatically by mysql_install_db). This script will create a new database named “innodb” with two tables in it – “table_stats” and “index_stats”. InnoDB will read/write the statistics from/to those tables. Second you must set the new parameter innodb_analyze_is_persistent to ON (it is OFF by default). This will instruct InnoDB to use the better (and slower!) sampling algorithm during ANALYZE TABLE and to save the results to the stats tables. To change the number of leaf pages that are being sampled during such a “persistent-results” ANALYZE TABLE set the parameter innodb_stats_persistent_sample_pages to a higher value.

Once the stats “meta” tables are present and stats for a given table exist in them, then those stats will be used whenever stats are needed – including open table/show table status/show index from and others. To stop using them you must DELETE the corresponding rows from innodb.table_stats and innodb.index_stats.

Beware of bugs

Some bugs in persistent stats code which may cause InnoDB to crash have been fixed, but are not included in 5.6.2. See:

http://bazaar.launchpad.net/~mysql/mysql-server/mysql-trunk/revision/3167.15.262

http://bazaar.launchpad.net/~mysql/mysql-server/mysql-trunk/revision/3167.15.266

http://bazaar.launchpad.net/~mysql/mysql-server/mysql-trunk/revision/3167.15.271

InnoDB estimates statistics for the query optimizer by picking random pages from an index. Upon detailed analysis, we found that the algorithm that picks random pages for estimation always picked the same page, thus producing the same result every time. This made it appear that the index cardinality was not updated by ANALYZE TABLE. Going deeper, the reason the algorithm always selected the same page was that the random number generator always generated numbers that, when divided by 3, always gave the same remainder (2).

The sampling algorithm selects a random leaf page by starting from the root page and then selecting a random record from it, descending into its child page and so on until it reaches a leaf page. In the particular case that was reported in the bug report, the root page contained only 3 records and the tree height was only 2 (i.e., the leaf pages were all just below the root page).

You can already guess what happened. The “random” numbers generated, not being so random, caused the algorithm to always pick the same record from the root page (the second one) and then descend to the leaf page below it. Every time. So, the 8 random pages that were sampled in order to get an estimate of the whole picture were in fact the same page, even in isolated ANALYZE TABLE runs.

So, clearly there was a problem with the random number generator. But why didn’t this problem seem to appear on 64-bit platforms? It would have, had we only enough time to wait. The random number generator, always generating numbers like 3k+2 of type unsigned long, at some point wrapped around 4 billion on 32-bit machines and started generating numbers like 3k+1. On 64-bit machines, where unsigned long is much bigger, this wrap did not occur. But it would have occurred if we ran the test for 1000 years!.

So, on 32-bit machines, at some point the first record from the root page was picked instead of the second one, and this caused some changes in the results produced by ANALYZE TABLE. Yet, on 64-bit machines, for all practical purposes, this “never” happens. By only looking at the symptoms, one would get the impression that the flaw existed only on 64-bit machines and that 32-bit systems were ok.

Well, what about the fix? A possible fix would be to change InnoDB in 64-bit environments to behave the same way it does in 32-bit environments. People who are used to the behavior of InnoDB on 32-bit machines and upgrade to a 64-bit machine might be satisfied, because the problem on 64-bit systems was “solved”. But in reality, this approach in no way would fix the underlying problem. The real solution is to replace the random number generator with a better one (fully realizing that algorithmic random number generators are only pseudo-random number generators).

Yet even that is not so simple. Making any change would have caused changes to index cardinality estimations, thereby causing changes in decisions made by the optimizer, resulting in different execution plans … and different, possibly worse, performance for queries. Because MySQL 5.1 and 5.0 are frozen for such drastic changes, we fixed this bug in the upcoming 1.0.4 release of the InnoDB Plugin.

In order to not break existing applications, and since many people wanted a fix for MySQL 5.0 and 5.1, we implemented this fix for MySQL under the control of a new configuration parameter (innodb_use_legacy_cardinality_algorithm), which is turned on by default, preserving past behavior. Because the “right fix” is to permanently change the random number generator, this new configuration parameter is not present in the InnoDB Plugin, and the “more random” random number generator will always be used.

And that is the end of the case of mistaken identity. It turns out that it is really hard to generate truly random numbers, hence the title of this blog post.

However, there are always technical and business issues to address. Given the low-level changes in the patch, was it technically sound? Was the patch stable and as rock solid as is the rest of InnoDB? Although it was written for the built-in InnoDB in MySQL 5.0.37, we needed to adapt it to the InnoDB Plugin. Could we make the patch portable to many platforms? Could we incorporate the patch without legal risk (so it could be licensed under both the GPL and commercial terms)?

Fortunately Google generously donated the patch to us under the BSD license, so there was no concern about intellectual property (so long as we properly acknowledged the contribution, which we do in an Appendix to the manual and in the source code). So, while the folks at Google are known for writing excellent code, we had to thoroughly review and test the patch before it could be incorporated in a release of InnoDB.

The patch improves performance by replacing InnoDB mutex and rw-mutex with atomic memory instructions. The first issue that arose was that the patch assigned the integer value -1 to a pthread_t variable, to refer to a neutral/non-existent thread identifier. This approach worked for Google because they use InnoDB solely on Linux. As it happens, pthread_t is defined as an integer on Linux.

But we had problems when the patch was tested on FreeBSD. We still needed to reference a non-existent thread, but in some environments (e.g. some versions of HPUX, Mac OS X) pthread_t is defined as a
structure, not an integer. As we looked at it, the problem became more complex. For this scheme to work, it must be possible to change thread identifiers atomically, using a Compare-And-Swap instruction. Otherwise, there will be subtle, mysterious and nasty failures.

We thought about enabling this patch only on Linux. But that approach was not optimal, because the patch could work perfectly well on other platforms where pthread_t is the same size as the machine word, like FreeBSD and Solaris, for example. We could have simply enumerated the operating system names where the patch would be supported, but this too was far from perfect. The ability to use the atomic instructions depends not only on the operating system, but the GCC version (has to be 4.1 or newer) and the capabilities of the CPU (some CPUs do not support Compare-And-Swap!).

So, we developed a dynamic compile-time check for checking whether the target environment (pthread_t type and size, GCC version, CPU capabilities) supports the use of atomic instructions as intended. This seemed to be the best approach, but then another problem arose!

Dynamic compile-time checks (i.e. autoconf) are part of the MySQL’s ./configure script. Innobase does not control this script, and so we must live with what is distributed by MySQL. To make things simple for
users, we wanted to avoid asking the users to re-create ./configure with the autotools. So we simply injected the required checks in the Makefile. This seemed to work fine. Only after release of the InnoDB Plugin 1.0.3 did a small crack in this approach arise. Fortunately this problem turned out to be easy to fix and only occurs if one compiles with “make -j” (in order to perform the make in parallel). See Bug#43740 for details and for a fix.

The bottom line is that sometimes more goes on “behind the scenes” than you might expect when it comes to incorporating a third-party contribution into a product. We are grateful for the Google patch, and are glad that we have been able to include it in the InnoDB Plugin in a way that maximizes its portability, while keeping things as simple as possible for users.