Well, it took us a little while (we’ve been busy !), but we’ve now posted our presentations on InnoDB from the MySQL Conference and Expo 2009. You can download these presentations by Heikki Tuuri, Ken Jacobs and Calvin Sun from the InnoDB website, as follows:

• Ken and Heikki: InnoDB: Innovative Technologies for Performance and Data Protection
• Heikki: Crash Recovery and Media Recovery in InnoDB
• Heikki: Concurrency Control: How it Really Works
• Calvin and Heikki: InnoDB File Formats and Source Code Structure

The description of these and other presentations about InnoDB are available here.

However, there are always technical and business issues to address. Given the low-level changes in the patch, was it technically sound? Was the patch stable and as rock solid as is the rest of InnoDB? Although it was written for the built-in InnoDB in MySQL 5.0.37, we needed to adapt it to the InnoDB Plugin. Could we make the patch portable to many platforms? Could we incorporate the patch without legal risk (so it could be licensed under both the GPL and commercial terms)?

Fortunately Google generously donated the patch to us under the BSD license, so there was no concern about intellectual property (so long as we properly acknowledged the contribution, which we do in an Appendix to the manual and in the source code). So, while the folks at Google are known for writing excellent code, we had to thoroughly review and test the patch before it could be incorporated in a release of InnoDB.

The patch improves performance by replacing InnoDB mutex and rw-mutex with atomic memory instructions. The first issue that arose was that the patch assigned the integer value -1 to a pthread_t variable, to refer to a neutral/non-existent thread identifier. This approach worked for Google because they use InnoDB solely on Linux. As it happens, pthread_t is defined as an integer on Linux.

But we had problems when the patch was tested on FreeBSD. We still needed to reference a non-existent thread, but in some environments (e.g. some versions of HPUX, Mac OS X) pthread_t is defined as a
structure, not an integer. As we looked at it, the problem became more complex. For this scheme to work, it must be possible to change thread identifiers atomically, using a Compare-And-Swap instruction. Otherwise, there will be subtle, mysterious and nasty failures.

We thought about enabling this patch only on Linux. But that approach was not optimal, because the patch could work perfectly well on other platforms where pthread_t is the same size as the machine word, like FreeBSD and Solaris, for example. We could have simply enumerated the operating system names where the patch would be supported, but this too was far from perfect. The ability to use the atomic instructions depends not only on the operating system, but the GCC version (has to be 4.1 or newer) and the capabilities of the CPU (some CPUs do not support Compare-And-Swap!).

So, we developed a dynamic compile-time check for checking whether the target environment (pthread_t type and size, GCC version, CPU capabilities) supports the use of atomic instructions as intended. This seemed to be the best approach, but then another problem arose!

Dynamic compile-time checks (i.e. autoconf) are part of the MySQL’s ./configure script. Innobase does not control this script, and so we must live with what is distributed by MySQL. To make things simple for
users, we wanted to avoid asking the users to re-create ./configure with the autotools. So we simply injected the required checks in the Makefile. This seemed to work fine. Only after release of the InnoDB Plugin 1.0.3 did a small crack in this approach arise. Fortunately this problem turned out to be easy to fix and only occurs if one compiles with “make -j” (in order to perform the make in parallel). See Bug#43740 for details and for a fix.

The bottom line is that sometimes more goes on “behind the scenes” than you might expect when it comes to incorporating a third-party contribution into a product. We are grateful for the Google patch, and are glad that we have been able to include it in the InnoDB Plugin in a way that maximizes its portability, while keeping things as simple as possible for users.

• Innovative Technologies for Performance and Data Protection by Ken and Heikki, Tues, 11:55am
• Crash Recovery and Media Recovery in InnoDB by Heikki, Wed, 2pm
• InnoDB Internals: InnoDB File Formats & Source Code Structure by Heikki and Calvin, Wed, 5:15pm
• Concurrency Control: How it Really Works by Heikki, Thurs, 2:50pm

Note the new times for the last two talks above. Be sure to check the conference schedule! Not much more to say about this topic, at least not here. Hear it all there!

• Enhanced concurrency & scalability: the “Google SMP patch” using atomic instructions for mutexing
• More efficient memory allocation: ability to use more scalable platform memory allocator
• Improved out-of-the-box scalability: unlimited concurrent thread execution by default
• Dynamic tuning: at run-time, enable or disable insert buffering and adaptive hash indexing

These new performance features can yield up to twice the throughput or more, depending on your workload, platform and other tuning considerations. In another post, we explore some details about these changes, but first, what do these enhancements mean for performance and scalability?

In brief, we’ve tested three different workloads (joins, DBT2 OLTP and a modified sysbench) using a memory-resident database. In all cases, the InnoDB Plugin scales significantly better than the built-in InnoDB in MySQL 5.1. And in some cases, the absolute level of performance is dramatically higher too! The charts below illustrate the kinds of performance gains we’ve measured with release 1.0.3 of the InnoDB Plugin. Your mileage may vary, of course. See the InnoDB website for all the details on these tests.

This release of the InnoDB Plugin incorporates a patch made by Ben Handy and Mark Callaghan at Google to improve multi-core scalability by using more efficient synchronization methods (mutexing and rw-locks) to reduce cpu utilization and contention. We’re grateful for this contribution, and you will be too!

Now to our test results …

Joins: The following chart shows the performance gains in performing joins, comparing the built-in InnoDB in MySQL (in blue) with the InnoDB Plugin 1.0.3 (in red).

As you can see from the blue bars in the above chart, with MySQL 5.1 using the built-in InnoDB, the total number of joins the system can execute declines as the number of concurrent users increases. In contrast, the InnoDB Plugin slightly improves performance even with one user, and maintains performance as the number of users rises. This performance improvement is due in large part to the use of atomics for mutexing in the InnoDB Plugin.

Transaction Processing (DBT2): The following chart illustrates a scalability improvement using the OLTP read/write DBT2 benchmark, again comparing the performance of the built-in InnoDB in MySQL with the performance of InnoDB Plugin 1.0.3.

Here, the InnoDB Plugin scales better than the built-in InnoDB from 16 to 32 users and produces about 12% more throughput with 64 concurrent users, as other bottlenecks are encountered or system capacity is reached. This improvement is likewise due primarily to the changes in mutexing.

Modified Sysbench: This test uses a version of the well-known sysbench workload, modified to include queries based on a secondary index, as suggested by Mark Callaghan of Google.

This time, the InnoDB Plugin shows significantly better scalability from 8 to 64 users than the built-in InnoDB in MySQL, yielding as much as 60% more throughput at 64 users. Like the previous examples, this improvement is largely due to the use of atomics for mutexing.

Modified Sysbench with tcmalloc: This test uses the same modified sysbench workload, but shows the difference between the built-in InnoDB (which uses the internal InnoDB memory allocator) and the InnoDB Plugin when using a more scalable memory allocator, in this case tcmalloc.

When the new configuration parameter innodb_use_sys_malloc is set to enable use of the memory allocator tcmalloc, the InnoDB Plugin really shines! Transaction throughput continues to scale, and the actual throughput with 64 users has nearly doubled!