• Plugin 1.0.7: 46min 21s
• Plugin 1.1: 32min 41s

Plugin 1.1 recovery is 1.5 times faster. Why would that happen? The numerous concurrency improvements in Plugin 1.1 and MySQL 5.5 can’t really affect the recovery. The honor goes to Native Asynchronous IO on Linux. Let’s try without it:

• Plugin 1.1 with –innodb-use-native-aio=0: 49min 07s

which is about the same as 1.0.7 time. My numerous other recovery runs showed that the random fluctuations account for 2-3min of a 30-45min test.

Why is native AIO good for you? Why is it better the  than the simulated AIO we already have? Here’s what Inaam Rana, our IO expert and the author of the AIO patch, says:

• During recovery typically redo log application is performed by the IO helper threads in the completion routine.
• With simulated aio the thread waits for IO to complete and then calls the completion routine.
• With native aio the thread doesn’t have to wait for io to complete, instead it picks a completed request and applies redo to it.

Read more about native AIO here.

You don’t have to do anything to take advantage of this feature. It is enabled by default and is used where available as determined by configure.

Here are some details about the test environment:

Hardware: HP DL480, 32G RAM, 2×4 core Intel(R) Xeon(R) CPU E5450  @ 3.00GHz, RAID5, about 1T total storage

Dataset:  1757549 dirty pages, 2808364565 bytes of redo. For the curious, it was a sysbench table with 400 million rows and the workload I used was random row update by a simple perl script.  Note that this is over 28G worth of dirty pages which means I had to use a very abusive settings of innodb_buffer_pool=29G and innodb_max_dirty_pages_pct=99, given only 32G of RAM. The recovery was done using the same settings and in the first few attempts the recovery would fail because of what was eventually diagnosed as bug 53122. As it happens, InnoDB recovery uses some memory outside of the buffer pool and it wanted more of it that was really necessary.

InnoDB configuration parameters:

–innodb-buffer-pool-size=28g
–innodb-log-file-size=2047m
–innodb-adaptive-flushing=0
–innodb-io-capacity=100
–innodb-additional-mem-pool-size=16m
–innodb-log-buffer-size=16m
–innodb-adaptive-hash-index=0
–innodb-flush-log-at-trx-commit=2
–innodb-max-dirty-pages-pct=99

This is highly artificial setup that targets maximizing the generation of dirty pages and redo, and using as much memory as possible for those dirty pages.

A good prep for the week of news is the article Introduction to MySQL 5.5, which includes information about the major performance and scalability features. That article will lead you into the MySQL 5.5 manual for general features and the InnoDB 1.1 manual for performance & scalability info.

Then there were the conference presentations from InnoDB team members, which continued the twin themes of performance and scalability:

• InnoDB: Status, Architecture, and New Features: Slides, Rate / leave feedback
• InnoDB Plugin: Performance Features and Benchmarks: Slides, Rate / leave feedback
• What’s New in MySQL 5.5? Performance/Scale Unleashed!: Slides, Rate / leave feedback
• What’s New in MySQL 5.5?: Performance and Scalability Benchmarks: Slides, Rate / leave feedback
• Introduction to InnoDB Monitoring System and Resource & Performance Tuning: Slides, Rate / leave feedback
• Backup Strategies with InnoDB Hot Backup: Slides, Rate / leave feedback

We hope that a good and useful time was had by all. Best regards to our European friends and colleagues whose return plans were disrupted by the Icelandic volcano!

With the plugin 1.1 release, InnoDB will have full support of Performance Schema, a new feature of MySQL 5.5 release. This allows a user to peak into some critical server synchronization events and obtain their usage statistics. On the other hand, in order to make a lot of sense of the instrumented result, you might need some understanding of InnoDB internals, especially in the area of synchronization with mutexes and rwlocks.

With this effort, the following four modules have been performance schema instrumented.

1. Mutex
2. RWLOCKs
3. File I/O
4. Thread

Almost all mutexes (42), rwlocks (10) and 6 types of threads are instrumented. Most mutex/rwlock instrumentations are turned on by default, a few of them are under special define. For File I/O, their statistics are categorized into Data, Log and Temp file I/O.

This blog is to give you a quick overview on this new machinery.

Start the MySQL Server with Performance Schema

To start with, you probably want to take a quick look at MySQL’s Performance Schema Manual (http://dev.mysql.com/doc/refman/5.5/en/performance-schema.html ), this gives you a quick overview on the general performance schema features.

The performance schema is by default built in with MySQL 5.5 release. However, you do need to add “-performance_schema” to your server boot command line or have performance_schema system variable enabled in your server configure file in order to enable the performance schema. Otherwise, it will be disabled.  Please note, you can specify “performance_schema” variable with no value or a value of 1 to enable it, or with a value of 0 to disable it.

When server starts, please pay attention to following lines in server error log:

“100407 16:13:02 [Note] Buffered information: Performance schema enabled.”

This means the server starts with performance schema running fine.

It could also display message such as:

“100407 16:13:02 [Note] Buffered information:  Performance schema disabled (reason: start parameters)”

This shows the performance schema is disabled due to lack of either “performance_schema” boot option or appropriate variable set in the configuration file.

The third type message would be “Performance schema disabled (reason: init failed)”, it is due to performance schema initialization failure (could due to reasons such as memory allocation failure etc.). This message is relatively rare. I have not encountered it. If you do hit it, please check  other performance schema related system variables, to see if they are out of reasonable range.

Performance Schema Database and its Tables

Assuming server starts fine with Performance Schema enabled, first stop you want to visit is probably the new database called “performance_schema”. All performance schema related tables are in this database:

mysql> use performance_schema

mysql> show tables;

Tables_in_performance_schema

These 18 tables can be categorized into a few big groups, such as “Instance” tables, “Wait” table with “History”, or “Wait” table with “Summary” and “Setup” table.

In the next few section, I will go through a few tables in this list that I think are important.

Find Instrumented Events with INSTANCE TABLES

To view what InnoDB events are active and being instrumented, please check following four “Instance” tables for corresponding modules:

MUTEX_INSTANCES
RWLOCK_INSTANCES
PROCESSLIST
FILE_INSTANCES

mysql> SELECT DISTINCT(NAME)
->  FROM MUTEX_INSTANCES
-> WHERE NAME LIKE “%innodb%”;

35 rows in set (0.00 sec)

Please notice there could be multiple instances of a mutex in the server,

mysql> SELECT COUNT(*)
-> FROM MUTEX_INSTANCES
-> WHERE NAME LIKE “%rseg_mutex%”;

1 row in set (0.92 sec)

This is why we need to use “SELECT DISTINCT (NAME)” clause in the initial query to get only the distinct mutex names from the MUTEX_INSTANCES table. Without the DISTINCT, there could be hundreds of instances of mutex being displayed. This also applies to other instance tables.

Also please note, if the mutex is not yet created, it will not be listed in the instance table, so you might see fewer events/instances than you might expected.

One last point for this section, buffer block mutex and rwlock are instrumented but disabled by default from performance schema instrumentation. The reason is that there comes one mutex/rwlock pair per 16k buffer block. Server with large buffer pool configuration could easily create thousands of instances of this type of mutexes/rwlocks.  This easily exceed the default value of max mutex/rwlock instances (1000) allowed. And user would require to extend the limit by setting system variable performance_schema_max_mutex_instances and/or performance_schema_max_rwlock_instances.
However, as we mentioned, the block mutex/rwlock are instrumented,  to enable them, you might need to change the code and un-define “PFS_SKIP_BUFFER_MUTEX_RWLOCK”.

Find out what is going on with EVENTS_WAITS_CURRENT table

The next table you might be interested in is the EVENTS_WAITS_CURRENT table,

mysql>  SELECT THREAD_ID, EVENT_NAME, SOURCE
->   FROM EVENTS_WAITS_CURRENT
->  WHERE EVENT_NAME LIKE  “%innodb%”;

4 rows in set (0.00 sec)

This table shows the latest instrumented activity for a particular thread. And the nice part of it is that it has the exact file name and line number of each event. So in case there is a hang/blocking situation (due to mutex/rwlock), you could know which mutex or rwlock is actually involved.

Check into “Limited history” with HISTORY tables

There are a couple of “HISTORY” tables that record each instrumented events. The EVENTS_WAITS_HISTORY table contains the most recent 10 events per thread. And EVENTS_WAITS_HISTORY_LONG contains the most recent 10,000 events by default.  They also come with the “SOURCE” field with file name and line number, and you might be able to do some aggregation on them to find some interesting behavior.

For example, following query gives you exact mutex instances that has been on the top list as shown in the history table:

mysql> SELECT EVENT_NAME, SUM(TIMER_WAIT), COUNT(*), SOURCE
-> FROM EVENTS_WAITS_HISTORY_LONG
-> WHERE EVENT_NAME LIKE “%innodb%”
-> GROUP BY SOURCE
-> ORDER BY SUM(TIMER_WAIT) DESC;

Or you can obtain the instance with the most average time wait:

mysql> SELECT EVENT_NAME, SUM(TIMER_WAIT)/count(*), source
-> FROM EVENTS_WAITS_HISTORY_LONG
-> WHERE EVENT_NAME LIKE “%innodb%”
-> GROUP BY source
-> ORDER BY SUM(TIMER_WAIT) / COUNT(*) DESC;

As mentioned, the history table has limited size, with 10 events per thread for  EVENTS_WAITS_and 10,000 for  EVENTS_WAITS_HISTORY_LONG.  However, you could extend the history length of these two tables by changing
“performance_schema_events_waits_history_size” and “performance_schema_events_waits_history_long_size” system variables. The performance_schema_events_waits_history_long_size can be extended to a million rows in maximum. However, please do not expect this would be enough. Even with 1 million events configured, in a busy system, it probably only contains a few seconds operation of the server.

Find out aggregated information from SUMMARY Tables

To get the overall aggregated value for these instances, you would need the “SUMMARY” table. There are 5 Summary tables,

EVENTS_WAITS_SUMMARY_BY_EVENT_NAME
EVENTS_WAITS_SUMMARY_BY_INSTANCE
EVENTS_WAITS_SUMMARY_BY_THREAD_BY_EVENT_NAME
FILE_SUMMARY_BY_EVENT_NAME
FILE_SUMMARY_BY_INSTANCE

As their name suggested, they are just events statistics aggregated with different criteria. Digging into these table gives you some idea where the contention could be.

For example, following query shows what is the hottest mutex (these values have unit as pico-second):

mysql> SELECT EVENT_NAME, COUNT_STAR,SUM_TIMER_WAIT,   AVG_TIMER_WAIT
-> FROM EVENTS_WAITS_SUMMARY_BY_EVENT_NAME
-> WHERE EVENT_NAME LIKE “%innodb%”
-> ORDER BY COUNT_STAR DESC;

This experiment shows the buf_pool_mutex has been the hottest mutex. However, aggregate on AVG_TIMER_WAIT shows that ibuf_mutex is the one we waited the longest, even though it is much less frequently accessed.

Also please note these tables can be truncated, so you can essentially reset these wait values, and start the counting and aggregation afresh.

Again, to really understand and interpret information from these tables require some understanding of internals that these mutexes/rwlocks reside. It could target to advanced users and developers who want to analyze performance bottlenecks. However, common users might still be able to infer certain information out of it, and have some creative usage of these statistics. For example, for some I/O bound servers, you might find double write buffer mutex is on the top listed mutexes in terms of total time waited. Then you might want to consider to turn off the double write buffer option etc.

Performance Impact:
The last item we discuss is that this performance schema comes with a cost. It does have visible performance impact. A simple dbt2 test with 50 warehouse, 32 connections on a server with 2G buffer pool size show about 8% performance impact with all performance schema events turned on. This is also confirmed by some sysbench’ tests.

In fact, to minimize performance impact, performance schema allows you turn off counting on individual event with SETUP tables,  you can use SETUP_CONSUMERS to turn on/off logging into history table etc., and SETUP_INSTRUMENTS to turn on/off counting on a particular mutex/rwlock etc.  However, turning off events counting cannot completely eliminate the performance impact from the performance schema. This is something the performance schema to be improved upon.

Summary:
In summary, we are providing a rich set of mutex, rwlock, I/O and thread usage information through Performance Schema instrumentations. It can be used to diagnose server performance bottlenecks, find possible hot spots in the server as well as gain a better understanding on system behavior/access pattern on modules where these mutexes/rwlocks reside in. However, it does come with a cost to server performance itself. So this is more suitable for development server tuning and studying. You might want to leave this out for any production server.

-Jimmy Yang

The original motivation behind this patch was the infamous Bug#26590 – MySQL does not allow more than 1023 open transactions. Actually the 1024 limit has to do with the number of concurrent update transactions that can run within InnoDB. Where does this magic number come from ? 1024 is the total number of UNDO log list slots on one rollback segment header page. And in the past InnoDB created just one rollback segment header page during database creation. This rollback segment header page is anchored in the system header page, there is space there for 128 rollback segments but only one was being created and used resulting in the 1024 limit. Each slot in the rollback segment header array comprises of {space_id, page_no}, where both space_id and page_no are of type uint32_t . Currently the space id is “unused” and always points to the system table space, which is tablespace 0. Now, onto the rollback segment header page. This page contains a rollback segment header (details of which are outside the scope of this blog entry ), followed by an array of 1024 UNDO slots. Each slot is the base node of a file based linked list of UNDO logs. Each node in this file based list contains UNDO log records, containing the data updated by a transaction. A single UNDO log node can contain UNDO entries from several different transactions.

Performance ramifications

When a transaction is started it is allocated a rollback segment to write its modifications. Multiple transactions can write to the same rollback segment but only one transaction is allowed to write to any one UNDO slot during its lifetime. This should make clear where the 1024 limit comes from. Each rollback segment is protected by its own mutex and when we have a single rollback segment this rollback segment mutex can become a high contention mutex.

Requirements

Backward compatibility in file formats is something we take very seriously at InnoDB.  InnoDB has always had the ability to use up to 128 pages but before this fix it created only one rollback segment. We had to figure out a way to make the multiple rollback segments change backward compatible, without breaking any assumptions in the code of older versions of InnoDB about absolute locations of system pages and  changes to system data. The 128 limit is a result of the latter. While there is space for 256 rollback segments, InnoDB uses only 7 bits from that field. Once we fix that we could in the future enable 256 rollback segments, however 128 seems to be sufficient for now. There are other scalability issues that need to be addressed first before 128K concurrent transactions will become an issue .

The solution

To keep backward compatibility the additional rollback segments are created after the double write buffer when creating the rollback segments in a new instance. We keep the total number of rollback segments set to 128 rather than 256 and set the remaining slots to NULL because the older versions will try and scan up to 256. This means that older versions of InnoDB could also benefit from these extra rollback segments.  If you decide to create the extra rollback segments with a newer version but decide to revert back to an older version the older versions should be able to use the additional rollback segments. Newer versions of InnoDB that contain this fix, will create the additional segments in existing instances only if the innodb_force_recovery flag is not set and the database instance was shutdown cleanly. The additional segments will be created by default when creating new instances.

As noted in the announcement, this release contains contributions from Sun Microsystems, Google and Percona, Inc., for which we are very appreciative. This page briefly describes each of the contributions and the way we treated them. The purpose of this post is to describe the general approach the InnoDB team takes toward third party contributions.

In principle, we appreciate third party contributions. However, we simply don’t have the resources to seriously evaluate every change that someone proposes, but when we do undertake to evaluate a patch, we have some clear criteria in mind:

• The patch has to be technically sound, reliable, and effective
• The change should fit with the architecture, and our overall plans and philosophy for InnoDB
• The contribution must be available to us under a suitable license

Let’s consider, in general terms, what these criteria mean in practice.

We have to expend a fair bit of effort to carefully evaluate and possibly modify a patch before we can include it in the release. Some of the third party contributions we’ve seen have not been portable, or have been developed just for Linux. It can take time to find an approach that enables a platform to take advantage of a new feature, even if the platform has the required capabililties. Some of the patches we’ve evaluated have contained actual bugs that would impact reliability, cause deadlocks or have other negative implications. InnoDB is a clean and elegant piece of code, yet some of its internal algorithms and behaviors are subtle and complex. Therefore, changes in the “guts” of InnoDB (or any storage engine) must be done carefully and thoroughly tested. Some patches that have been offered make a difference, but only when compared to an inappropriate “baseline”. At any given point, we would look to include a patch only if it makes a significant improvement over the “best” version or configuration of InnoDB available at the time. We like to test each patch in isolation, to assess its individual value. This requires some rigorous performance testing, with multiple workloads.

From time to time, third parties have made suggestions for changes that may seem attractive at first, but don’t make sense longer term. In general, we may have a more comprehensive approach to a problem or requirement that we would like to implement, rather than incorporate a patch that would introduce a feature that would ultimately be made obsolete. We prefer to have fewer “knobs” and tuning complexity, so we’re more inclined to implement heuristic, self-tuning capabilities than we are to add new configuration parameters. Lastly, we take care to protect the ability to upgrade and downgrade user databases with the file format management features in the InnoDB Plugin. If a patch requires an on-disk change, we will defer its incorporation until the time comes to implement a new file format.

For us to be able to make continued investment in InnoDB, we must be able to license the software commercially. OEMs and ISVs who incorporate MySQL with InnoDB in their products may not wish to release their products in open source form. Therefore, for each contribution we are to accept, we must have clear legal rights to the change.

Beyond all that, of course, we take care to carefully document each new feature, both in terms of form and function. We try hard to explain the implications of a feature, providing information about what it does, and when and where to use a feature, as well as how to do so. And, we generally speaking are committed to upward compatibility and support of a feature once it is introduced.

It’s pretty clear that the integrity of InnoDB, with its broad adoption and importance everywhere it is used, is paramount to you and to us. You can trust the InnoDB team to protect InnoDB now and in the future, while being open to suggestions and contributions. Let us know if you think we’re doing a good job!

Well, it took us a little while (we’ve been busy !), but we’ve now posted our presentations on InnoDB from the MySQL Conference and Expo 2009. You can download these presentations by Heikki Tuuri, Ken Jacobs and Calvin Sun from the InnoDB website, as follows:

• Ken and Heikki: InnoDB: Innovative Technologies for Performance and Data Protection
• Heikki: Crash Recovery and Media Recovery in InnoDB
• Heikki: Concurrency Control: How it Really Works
• Calvin and Heikki: InnoDB File Formats and Source Code Structure

The description of these and other presentations about InnoDB are available here.

InnoDB estimates statistics for the query optimizer by picking random pages from an index. Upon detailed analysis, we found that the algorithm that picks random pages for estimation always picked the same page, thus producing the same result every time. This made it appear that the index cardinality was not updated by ANALYZE TABLE. Going deeper, the reason the algorithm always selected the same page was that the random number generator always generated numbers that, when divided by 3, always gave the same remainder (2).

The sampling algorithm selects a random leaf page by starting from the root page and then selecting a random record from it, descending into its child page and so on until it reaches a leaf page. In the particular case that was reported in the bug report, the root page contained only 3 records and the tree height was only 2 (i.e., the leaf pages were all just below the root page).

You can already guess what happened. The “random” numbers generated, not being so random, caused the algorithm to always pick the same record from the root page (the second one) and then descend to the leaf page below it. Every time. So, the 8 random pages that were sampled in order to get an estimate of the whole picture were in fact the same page, even in isolated ANALYZE TABLE runs.

So, clearly there was a problem with the random number generator. But why didn’t this problem seem to appear on 64-bit platforms? It would have, had we only enough time to wait. The random number generator, always generating numbers like 3k+2 of type unsigned long, at some point wrapped around 4 billion on 32-bit machines and started generating numbers like 3k+1. On 64-bit machines, where unsigned long is much bigger, this wrap did not occur. But it would have occurred if we ran the test for 1000 years!.

So, on 32-bit machines, at some point the first record from the root page was picked instead of the second one, and this caused some changes in the results produced by ANALYZE TABLE. Yet, on 64-bit machines, for all practical purposes, this “never” happens. By only looking at the symptoms, one would get the impression that the flaw existed only on 64-bit machines and that 32-bit systems were ok.

Well, what about the fix? A possible fix would be to change InnoDB in 64-bit environments to behave the same way it does in 32-bit environments. People who are used to the behavior of InnoDB on 32-bit machines and upgrade to a 64-bit machine might be satisfied, because the problem on 64-bit systems was “solved”. But in reality, this approach in no way would fix the underlying problem. The real solution is to replace the random number generator with a better one (fully realizing that algorithmic random number generators are only pseudo-random number generators).

Yet even that is not so simple. Making any change would have caused changes to index cardinality estimations, thereby causing changes in decisions made by the optimizer, resulting in different execution plans … and different, possibly worse, performance for queries. Because MySQL 5.1 and 5.0 are frozen for such drastic changes, we fixed this bug in the upcoming 1.0.4 release of the InnoDB Plugin.

In order to not break existing applications, and since many people wanted a fix for MySQL 5.0 and 5.1, we implemented this fix for MySQL under the control of a new configuration parameter (innodb_use_legacy_cardinality_algorithm), which is turned on by default, preserving past behavior. Because the “right fix” is to permanently change the random number generator, this new configuration parameter is not present in the InnoDB Plugin, and the “more random” random number generator will always be used.

And that is the end of the case of mistaken identity. It turns out that it is really hard to generate truly random numbers, hence the title of this blog post.