1. 7 Approved Caching Technologies
2. Caching Strategies

Our goal in the previous few weeks has been to identify caching strategies and technologies. We have now used this knowledge to identify where to invest our focus on improvements. Our conclusion is to go fiercely ahead investigating the following couple of topics further.

1. Reverse proxy caching (Varnish and ESI)

Håkon and Michael will be putting in the effort to research what is possible to do with ESI and VCL.  By getting an overview on how we best can utilize Varnish in our applications we can either learn how to configure Varnish ourselves, or inform hosting providers of concrete scenarios which then they can configure for us. It’s worth noting that Varnish has been performing flawlessly on almost every installation we have had to date, so this is an important tool. Our hope is to develop some sensible and helpful best practices for using Varnish with and without ESI.

2. Data caching

Lars and Stefan will be looking into how we can improve our very own AFWCache-mechanisms (AFW is our inhouse, still unreleased framework). We might include MySQL / SQLite data-caching options, as well. Another core topic for them to explore further is whether we’ll be extending the use of APC to also include data caching (as opposed to only opcode caching). In order to decide to use APC for data caching on high performance production environments, we have to learn more about it’s behavior under duress, and we need to know if we can make it degrade gracefully (i.e what happens if APC runs out of memory?)

We will currently not invest more energy in

View and subview caching (application caching) which basically is the slower sibling of Varnish with and without ESI. Although more flexible (you can process cookies), we choose to discard any effort in improving in this area to keep our focus where it need be for the moment. Also our current support for view and subview caching through data caching is performing just fine.

Query Cache – We are requiring query cache to be set to “ON” on every installation. Our conclusion is that this is more than enough for our current needs. We’ll be revisiting the topic of setting query cache to DEMAND and using the /*SQL_CACHE*/ trigger when we are making the new Twitter, or something with a similar requirement for database scaling.

Client caching – We will be refining our guidelines and best practices for setting the correct headers, but we will not invest in the topic of Local Storage (“the new cookie”) as of yet. We’ll wait until the browser support is broader. Local storage is expected to save us a lot of AJAX-callbacks in the future. More on this sometime in the future.

Opcode cache – Having discarded a couple of other candidates, we require APC for PHP on all our installations. Once up and running, it just works without any intervention, so we’ll not be investing any more energy into the topic of opcode caching. We have strengthened our systems setup testing for APC, and we’ll leave it at that.

Have a look at VG Meta (Norwegian only)

(Bookmark this page, the graph updates during the day!)

During the weekends the graph is normally flat, so check back in during weekdays. First cup is usually around 7 or 8 am.

Edit: More graphs are now available

Week

Month

Year

We wanted to honor Rick on his special day, so we rasterbated him up on our office wall. If you drop by to celebrate with us we’ll hook you up with a beautiful song and a beer.

So that’s what we are were doing.

We’re even changing our home-page to reflect this. Visit http://aptoma.com/index_jul.php to experience true Christmas spirits in Aptoma. Don’t mind the wine and beer.

Merry Christmas.

Nested updates

When INSERTing, REPLACEing, UPDATEing or DELETEing several rows, nesting them together in a single long query is faster, preferably with INSERT DELAYED where applicable. The nested queries are faster in them self since the query is only interpreted once and only needs to update indexes once whereas several single rows inserted require one update each. When considering this remember that on a MyISAM table with a more even read/write load, the higher overhead of multiple fast queries might be better than a rare longer lasting table lock. UPDATEs can’t be nested, but can be written as a temporary INSERT followed by an UPDATE.

/* CREATE TABLE */
CREATE TABLE associates ( id INT, value INT, name VARCHAR(10) );
CREATE TABLE tmp_associates ( id INT, value INT, name VARCHAR(10) );

/* INSERT */
INSERT DELAYED INTO associates SET ( value, name ) VALUES
		( 4, 'Mr Pink' ), ( 2, 'Mr White' ),
		( 4, 'Mr Orange' ), ( 6, 'Mr Brown' );

/* UPDATE */
TRUNCATE TABLE tmp_associates;
INSERT INTO tmp_associates VALUES ( 3000, 'Mr Pink' ),
		( 22, 'Mr White' ), ( 11, 'Mr Orange' ),
		( 91, 'Mr Brown' );
UPDATE associates, tmp_associates
		SET associates.value = tmp_associates.value
		WHERE tmp_associates.name = associates.name;

Subselects

In most cases, don’t use them, use JOINs. The foreachesque way of thinking that leads to subselects is logical and tempting to use, but as most RDBs work in sets, they are in most cases dead slow compared to JOINs. It should be noted that in some cases subselects can be hundred of times faster, but those cases are special and you should really know what you are doing before opting for it. Of all the JOINs (inner, outer, left, right, cross and more) STRAIGHT_JOIN is one you shouldn’t use. STRAIGHT_JOIN tells the database to always join in the order that you specify and to not use its optimizer. Have we learned anything from Hollywood it’s that computers (and dirty apes) are smarter than humans, don’t flatter yourself by thinking otherwise. :)

Boolean flags

Apparently Torfinn here at Aptoma did research into this some time ago and discovered that NULL values are much slower compared to the number zero and allows nicer query-syntax. In MySQL TINYINT, BOOL and BOOLEAN are all synonyms for the same single-byte data type so it doesn’t matter which one you choose, but TINYINT with the values 1 and 0 are more logical and easier to understand for someone who doesn’t know this. The CREATE string for avoiding NULL values is like this:

CREATE TABLE t ( deleted TINYINT NOT NULL DEFAULT 0 );

The conditional part of a query can then be written like this:

SELECT id FROM t WHERE !deleted;

Temp tables

If you feel the write rate of your table is too high for optimal query cache usage and you don’t have any rush inserting new rows, consider inserting them into a temp table and at set intervals INSERT INTO the main table. This leads to faster initial INSERTs, potentially more query cache hits and less overhead updating indexes.

Do more once, not some foreach

In normal programming languages foreach is great, but for SQL it’s seldom optimal. If you end up doing a query foreach something, look into doing two foreachs in PHP and one larger SQL-query. One foreach to gather conditional data (build a string suitable for an IN condition) and one after the query is executed where you first locate the row(s) you need. Ordering the data in the same order in both PHP and SQL will save you lots power as you can read and match rows from the returned data set sequentially.

I’ll provide an example. The previous paragraph could be a bit hard to grasp. In this first function a date restricted query is executed foreach user looking for its name. In the second function all user names are added to an IN() function and a single query gathers the data for all listed users within the same date restriction and is ordered by user name. Then some PHP orders the users in the same order ( hopefully — be careful about collation and special chars) and foreach user locate matching row in the result set.

The bad, with potentially a bunch of queries:

function weekReport() {
	$lusers = array( 'Joe', 'Frank', 'Bill', 'Ted' );
	$errors = array();
	$today = date( 'Y-m-d 00:00:00' );
	$sql = "SELECT SUM( errors ) AS errors FROM problems WHERE
	created < '{$today}'
		AND CREATED >=
		DATE_SUB( '{$today}', INTERVAL 7 DAY ) AND name = ";
	foreach ( $lusers as $luser ) {
		$result = mysql_query( $sql . "'{$luser}';";
		if ( $row = mysql_fetch_assoc( $result ) ) {
			$errors[$luser] = $row['errors'];
		}
	}
	return $errors;
}

The potentially good way, with one query:

function weekReport() {
	$lusers = array( 'Joe', 'Frank', 'Bill', 'Ted' );
	$errors = array();
	$today = date( 'Y-m-d 00:00:00' );
	$sql = "SELECT name, SUM( errors ) AS errors
		FROM problems WHERE
		created <  '{$today}'
		AND CREATED >=
		DATE_SUB( '{$today}', INTERVAL 7 DAY )
		AND name
		IN ( '" .  implode( '\',\'', $lusers ) . '\'' . ' )
		GROUP BY name, ORDER BY name DESC;';
	$result = mysql_query( $sql );
	$lusers = array_sort( $lusers );
	$found = true;
	$row = false;
	foreach ( $lusers as $luser ) {
		if ( $found ) {
			$found = false;
			$row = mysql_fetch_assoc( $result );
		}
		if ( $row && $row['name'] == $luser ) {
			$found = true;
			$errors[$luser] = $row['errors'];
		}
	}
	return $errors;
}

Configuration

The default cache sizes aren’t very performance tuned so you probably want to increase them if you have the resources and access to do so.

SHOW VARIABLES;

will show current MySQL settings and these are some of the more important ones:

When using InnoDB
innodb_buffer_pool_size – Total cache size for InnoDB databases. Default 8 MB and can be set to up to 50-80% of total system memory when only using InnoDB tables.
innodb_log_buffer_size – How much log to buffer in memory before writing to disk. Increasing this could give a performance boost.

When using MyISAM
key_buffer_size – Total cache size for MyISAM databases. Default 8 MB and can be set to up to 50-80% of total system memory when only using MyISAM tables.

Query Cache
query_cache_size – Total cache size for query results. Must be set to more than 0 byte and a few megabyte should be enough in most cases.
query_cache_type – Query Cache status. 0 or OFF is deactivated, 1 or ON is always on and 2 or DEMAND is on when using key word SQL_CACHE. ON is the easiest as no modification to the SQL statements are required but will decrease performance on cache misses. DEMAND won’t have the same performance hit on all queries and could be the best choice when all programmers know each tables access levels and know when Query Cache will be needed and when not.

When using MEMORY
max_heap_table_size – Total data size for MEMORY tables.

Helpful commands

Here in the end I will just list a few commands you should know. Search the MySQL manual to see what they do.

SHOW VARIABLES;
SHOW PROCESSLIST;
SHOW FULL PROCESSLIST;
SHOW STATUS;
FLUSH STATUS;
SHOW INNODB STATUS;
SHOW TABLE STATUS table;
SHOW CREATE TABLE table;
SHOW INDEXES FROM table;
SHOW KEYS FROM table;
OPTIMIZE TABLE table;
ANALYZE TABLE table;

Presentations

MySQL Performance  Blog from the authors of High Performance MySQL has several good presentations worth reading. In some of them you need to read between the lines ( as they for the most part consist of bullet points ), but that should be easy enough.
http://www.mysqlperformanceblog.com/mysql-performance-presentations/

Natural keys versus surrogate keys

Don’t always add an auto_increment integer as a key when a naturally unique field ( or a group of fields ) exists of reasonable size. The best example of this a join-table where you normally have two primary keys from some other tables where all SELECT queries will be conditional on them. When creating multi-field primary keys, you should also have in mind the order in which you add them. Have the field you want to do most queries on first so the index can be used. If both fields are equally important, also add an index on the other field.

COUNT(*)

When using MyISAM, COUNT(*) needs almost no processing as the primary key index knows exactly how many primary keys there are, and the query is very fast. But the index in an InnoDB table does not have this feature, (because of the Multi Version Concurrency Control, MVCC) and one of the way it finds out how many rows there are is to sequentially read them all, an operation potentially tens of thousand times slower. There are a few alternatives and the first you should do is to decide how exact you need the the number to be. The best solution when it has to be exact is to add ON INSERT and ON DELETE triggers to the table and increment/decrement an integer in some other table. That table could use the MEMORY engine for optimal speed. Other solutions could be to manually increment/decrement a value in Memcached on INSERT and DELETE queries or have a cron-job execute a COUNT(*) once in a while and cache the value. If one row is INSERTed per day or several rows are INSERTed once a day at a specific time, then there is no point in counting the rows more than than once a day, preferably right after the INSERTs have finished.
Insted of

SELECT COUNT(*) FROM wins WHERE player_id = 123;

to find out how many wins a player has, why not just have ‘wins‘ as a field in the users table and increment when the player is winner?

Don’t use functions on indexed fields

When using indexes on a field its also important to not use SQL-functions on the field but list it unchanged alone on one side of the equality sign. In this example only the last will use the index on the field date:

SELECT date FROM table WHERE UNIX_TIMESTAMP( date ) = 1057941242;
SELECT date FROM table WHERE date = FROM_UNIXTIME( 1057941242 );

UNIQUE and LIMIT 1

When a field will be unique but is not suitable as primary key, UNIQUE could be a good choice for it. This is because UNIQUE is in fact treated the same way as the primary key and the optimizer will know that only one row will match. Both lookup times and memory allocation will probably be affected by this, and some of the same performance boost can be had by applying LIMIT 1 to queries where only one row is updated, such as:

UPDATE users SET kickass = 1 WHERE
name = 'nameless space marine' LIMIT 1;

INSERT DELAYED

MyISAM supports INSERT queries where index updating is held off until other queries with higher priority has completed. The performance boost is supposed to be significant in high load situations, but the chance of out-of-sync indexes increases where a slow REPAIR TABLE and an index rebuild could be needed after a system crash.
http://dev.mysql.com/doc/refman/5.0/en/insert-delayed.html

Slim is fast

Keep all fields as trim as possible. Use unsigned numbers to double the range when only positive numbers are needed and use TINYINT, SMALLINT, MEDIUMINT and limited VARCHARS as much as possible. You might need INT as AUTO_INCREMENT primary key on the users table, but an unsigned TINYINT ( 0-255 ) could be plenty for the categories table. Its a bit counter-intuitive, but splitting nullable-, text-, blob- and less accessed fields into a one-to-one sister table could give you a performance boost. This way you can faster and with less memory load more rows of the more accessed part of the table.
http://dev.mysql.com/doc/refman/5.0/en/data-types.html

Update internal statistics

There are commands which will update cardinality and indexes in a table. For MyISAM the command is OPTIMIZE TABLE and for InnoDB it’s ANALYZE TABLE. Its worth mentioning that OPTIMIZE TABLE will issue a write lock on the table and block all other queries until its done.
http://dev.mysql.com/doc/refman/5.0/en/optimize-table.html
http://dev.mysql.com/doc/refman/5.0/en/analyze-table.html

mysqlhotcopy

When doing a database dump on a large MyISAM table, consider using mysqlhotcopy instead of mysqldump.
http://dev.mysql.com/doc/refman/5.0/en/mysqlhotcopy.html

Query cache

By default MySQLs query cache is disabled and supports three modes: OFF, ON and DEMAND. OFF prevents caching or retrieval of any results, ON caches/retrieves as much as possible except statements that begin width SELECT SQL_NO_CACHE. DEMAND only caches queries starting with SELECT SQL_CACHE. Query cache is between 200 and (infinity+1)% faster on a cache hit, but any miss is normally 5-25% slower. However, both these numbers can vary a lot. On a table with a high number of reads with a high probability of identical queries, query cache will probably increase performance significantly, but since any INSERT, DELETE or UPDATE will invalidate all cache on that table, performance on a table with frequent writes is likely to drop. What’s also important is that the subsequent queries must be identical to the first for its cache to be used. That includes case, white space and brackets. When actively using query cache on tables with dates it’s also important to write reusable and cacheable queries. Any use of NOW() or equivalent non-deterministic functions will render query cache unused even within the same second, so date and time should be calculated outside of MySQL. When doing so, first analyze the need for an exact query. If you can get away with < 1 minute accuracy, don’t add date( ‘Y-m-d H:i:s’ ) but date( ‘Y-m-d H:i:00′ ) so any identical queries within the same minute will get the cached result.
http://dev.mysql.com/doc/refman/5.0/en/query-cache.html

SELECT * ©

You don’t need everything so don’t ask for it! It will only slow down your query, increase memory usage while sorting and make the query less understandable when running ‘SHOW PROCESSLIST’.

GROUP BY or DISTINCT

Both these can be used with the same result but performance-wise, there is a difference. For simpler queries, DISTINCT is often faster but in more advanced queries GROUP BY could be the best choice. Benchmark your queries to find out which one to choose.

IP addresses

As we normally see IP addresses in the form of 000.000.000.000, VARCHAR( 15 ) might be natural to use, but you should treat it as a number because it will both take up less space and could be faster and easier to query. The data type of choice for an IPv4 address is the four byte unsigned INT. You then use INET_ATON and INET_NTOA to convert that number to the dotted-quad representation we normally use. (Or keep it as an int, it will always be superior to the string representation! )

INSERT INTO blacklist SET ( ip, date ) VALUES
( INET_ATON( '209.207.224.40' ), NOW() );

SELECT date FROM blacklist WHERE ip = INET_ATON( '209.207.224.40' );
SELECT INET_NTOA( ip ) FROM blacklist WHERE ip >= INET_ATON( '127.0' )
  AND ip < INET_ATON( '128.1' );

The next part will be released next week. Stay tuned with rss.

Indexes are essential for fast and effective conditional queries and you should always use them. If you can afford the extra storage space, that is. Selecting the right indexes is both a space vs. speed consideration, and the result of an analysis of the queries you will be doing. Therefore it shouldn’t be a set-and-forget operation but revised as queries are written and then again revised after live application operation has started, when real-world usage and data can be analyzed. Great tools for this are MySQL’s slow query log, the EXPLAIN command and query rate counters, all of which are commented in this post.

The different engines handle indexes in different ways. Thus indexes should be revised when switching engines. One serious issue is always there though: MySQL will only use one index in a query! (except when performing UNIONs) The results are that adding an index per field could be a huge waste of space and processing power if that index is never used. Another important note is that the order of the secondary index fields matters. If field one and three in an index is used in the query only field one will be index-searched because field three requires that field two is also used.

http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html

The grey smoke ( how they work )

Warning : The following section is a crude explanation of how indexes work. The different engines have each their own way of implementing indexes. This section should only be read as a way they could work.

Indexes are ordered duplicates of selected fields from a table. Since these tables are ordered, MySQL can find matching rows in a WHERE clause much faster than in the random order of the original table. An index can also include several fields where the data is stored as a tree (in most cases) in the order you set the fields. The first field becomes the root node and following fields becomes the branches.Here is an example of a table with the fields id, first name and family name.

1,	Alec,		Baldwin
2,	Adam,		Baldwin
3,	William,	Baldwin
4,	Stephen,	Baldwin
5,	Albert,		Einstein
6,	Adam,		West

When executing the query :
SELECT family FROM people WHERE first = ‘William’;
MySQL would have to read all the first names to find the matches. But if you add the index ( first ), MySQL would create an index looking something like :

Adam,		2
Adam, 		6
Albert,	 	5
Alec,		1
Stephen, 	4
William,	3

The query would then by different kinds of magic move almost directly to the ‘William’ section and find the references to the rows in the original table.

If we then add the index ( family ) this query would also go faster as the “family“-index would be used :

SELECT first FROM people WHERE family = ‘Einstein’;

But when executing a query with WHERE-clauses on both the family and first fields only one will be used and probably the one with highest cardinality (read about it further down). The following query would probably use the “first” index and then exclude “West” from the list after retrieving all the Adams from the original table.

SELECT first, family FROM people WHERE family = ‘Baldwin’ AND first = ‘Adam’;

So what we want is an index with both fields like ( first, family ). MySQL will then create a tree in this shape:

Adam,		Baldwin,	2
		West,		6
Albert,		Einstein,	5
Alec,		Baldwin, 	1
Stephen,	Baldwin,	4
William,	Baldwin,	3

The previous query would then be able to exclude rows based on both conditions an find the matching table references. (You would also have a covering index. More on this later.)

A very important thing to know is the difference between ( first, family ) and ( family, first ) as the latter would look like this:

Baldwin,	Adam,		2
 		Alec,		1
 		Stephen,	4
 		William,	3
 Einstein,	Albert,		5
 West,		Adam,		6

Index prefix compression could also be mentioned here. Again the exact implementations could be very different from my version but you should be able to get the gist of it. When the start of multiple index fields match the engine could do some magic and only store that bit once. This can save a lot of space and adding fields with very low cardinality to an index could increase index size minimally, but not all engines implement them. Let’s add a few people and the field “banana” to our table and see how it could look.

First without compression:

Baldwin,	Adam,		Yellow,		2
 		Adam,		Yellow,		9
 		Adam,		Brown,		10
 		Adaminium,	Yellow,		7
 		Adamsomething	Brown,		8
 		Alec,		Yellow,		1
 		Stephen,	Yellow,		4
 		William,	Yellow,		3
 Einstein,	Albert,		Yellow,		5
 West,		Adam,		Yellow,		6

Then with compression:

Baldwin,	(Adam),		(Yellow),	2
 		(),		(),		9
 		(),		Brown		10
 		()inium,	Yellow		7
 		()something	Brown		8
 		Alec,		(Yellow),	1
 		Stephen,	(),		4
 		William,	(),		3
 Einstein,	Albert,		(),		5
 West,		Adam,		(),		6

I hope this explanation will help understanding why indexes are so usefull but still why a set-and-forget strategy seldom works. If 90% of your heavy queries can’t use your long and space consuming indexes then it’s just slowing you down.

The Slow Query Log

The variable long_query_time holds the number of seconds a query can be executed before its added to the slow log. Default value is 10 seconds. It’s recommended to use the mysqldumpslow to summarize the queries.

http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html

EXPLAIN

Added in front of any query, a summary of MySQL’s optimization engine’s choices for the query are shown. This is an essential and simple way to check that your queries work with indexes as planned. The important fields are type, key, keylen, rows and extra. Type will tell you how the data is read where ALL is your mortal enemy and const and range your more likely targets. There are several others though, google it search them up with your favorite search engine. The key field tells you what key the optimizer chose, if any. On any large data set you would probably want an index here, but even if there are indexes available, MySQL might choose not to use them. If you see that MySQL does poor index choices on a query, USE INDEX or FORCE INDEX hints can be added to the query, but is not recommended for other than testing purposes as the data set is likely to change and the dynamic query path is safer in the long run. keylen tells you how much of the key is used in bytes. A small number such as 3 could indicate that just the first MEDIUMINT field in the index is used while 200 could be the result of a good multi field hit. (Unless your text indexes are unnecessarily long, read about partial indexes below). Rows is the number of hits the optimizer expects to find based on its table statistics. In the Extra field you should see “using index, using where” for a fast query but “using filesort” and a few other could mean problems. Again, google it what I wrote earlier.

http://dev.mysql.com/doc/refman/5.0/en/explain.html

Query rate counter

D-I-Y by just incrementing an integer in Memcached or a MEMORY table every time your code does a database query or by adding a trigger. After some usage you can analyze the data and see which queries you should optimize for. Its really up to you how you choose to implement it.

http://dev.mysql.com/doc/refman/5.0/en/triggers.html

Cardinality

When doing a SHOW INDEXES FROM table; cardinality is one of the more important fields to inspect. Cardinality is the uniqueness of the index so far and the higher cardinality a field has, the better it will work as an index. Where the primary key has a cardinality equal to COUNT(*), the TINYINT(1) field deleted (boolean one or zero for true or false) would have a cardinality of maximum two, regardless of how many rows there are. A primary key lookup often goes at optimal speed but in the case of an index on the TINYINT(1) deleted field, won’t MySQL in most cases even use the index. This is because when the optimizer engine predicts a hit percentage of more than about 30% it will opt for the often faster sequential read of the entire data set. You should therefore order the fields in multiple field indexes first by field hit frequency (how often that field is part of the WHERE clause of your queries), and then by cardinality. This excludes as many rows as possible early in the query and to gives the optimizer an easier and more predictable job. Sometimes, however, even if the cardinality of a field is sub-optimal it’s better to include it in your index just so you can have a covering index.

Covering index

When processing an indexed query, MySQL first reads the index file, excludes as many rows as possible and then uses the byte offset (MyISAM) or primary key (InnoDB) to read the rest of the data needed in those rows from the data file. In some cases, the second file read could increase the latency more than if a larger data set was just read from the index. An index including all fields in a query where just one file read is needed is called a covering index. The decision to having covering indexes is then again a space/speed consideration.

Partial indexes

Normal indexes on text fields (TEXT, VARCHAR, etc) isn’t always a good choice because each distinct index in a VARCHAR(100) will occupy a lot of unnecessary space. With a partial index, only a set number of chars are indexed starting at the beginning of the field. The VARCHAR field for storing a persons first name should be wide enough for ‘Dominar Rygel XVI’ to store his name, but an index of the first three to six chars is probably enough to get a good enough cardinality on most first names to exclude >95% of the rows. How many chars are needed can easily be checked by just adding a new index and compare the listed cardinality after updating the table statistics. At 10 000 users from Scandinavia you might need five chars for a >95% cardinality, but at 100 000 users from all over the world, four might result in the same cardinality with just marginally better result at five.

Email domain queries with index

In some situations you need to query email address by domain name (no you won’t, but you might want to learn this trick anyway) and instead of the slow LIKE %domain.com with a query type of ALL, the solution could be to reverse the address and add a partial index since indexes work from left to right. The address could be reversed in PHP with strrev() or by MySQL with REVERSE(), but a better solution could be to use the triggers BEFORE INSERT and BEFORE UPDATE, but that’s up to you. The query would then end up like this:

SELECT name FROM customers WHERE email LIKE 'on.amotpa%';

or

SELECT name FROM customers WHERE email LIKE REVERSE( '%aptoma.no' );

and an insert could look like this

INSERT INTO customers ( name, email ) VALUES ( 'Kryten', REVERSE( 'mechanoid4000@aptoma.no' ) );

Realistic data set when testing

One of the reasons why indexes should be revised over time is that MySQL uses dynamic execution paths based on statistics updated on data change. If you only have one row in a table, all the indexes in the world (except when querying with FORCE INDEX, but lets forget about that ancient religion) won’t make MySQL use them, because it knows that a sequential read of the entire datafile will be just as fast or faster, but with multi-million rows a sequential read (a query type ALL) is probably the last thing it wants to do. Cardinality is also important to have in mind when testing as indexes again won’t be used if your multi-million rows of test data are all identical. The solution is to either make a more advanced random data creator, manually add realistic data or export realistic data from some other live database.

Moral of the story

If you want the step up from slow to reasonable performance and beyond using MySQL, indexes are mandatory but be weary of their size. You might want to add indexes in all colors and shapes available but rembember that the data is duplicated for each index and when you INSERT, UPDATE or DELETE all indexes have to be updated as well as the main table. Make the index a short as you can while keeping the cardinality as high as possible, have multiple field indexes that match as many of your queries as possible ( rewrite the non index using queries with more constraints than needed if you have to/can ) and make sure the data types of your fields are as trim as possible. The last part will also come up in my next post. Stay tuned for more of my lies next week and please comment on any errors in the post or if you by some miracle learned anything from it. :)

One of MySQL’s more powerful features is its support for multiple different database engines and the first thing you should do when creating a table is to select the storage engine best suitable for your needs. In MySQL 5.1 there are three engines worth mentioning, MyISAM, InnoDB and MEMORY but new interesting engines are in development such as Falcon, Maria, PBXT and solidDB.
http://solutions.mysql.com/engines.html

MyISAM

MyISAM is the default engine of MySQL and a good choice when your access levels are >90% read or >90% write on a high load system or in any case on a low-load system. Both SELECTs and INSERTs on MyISAM tables can be incredible fast, but where it falls through is how it handles locks. The problem is that table-level read/write locks are the only ones supported. This means that when a UPDATE or DELETE operation is executed all other queries are put on hold until completed, and when a INSERT operation is executed all other INSERTs are on hold. ( Note: this means that several read operations can be executed simultaneously. I’ll discuss more on INSERT DELAYED in a future post.) This could result in multi-second or even multi-minute delays on SELECTs when the SELECT/INSERT ratio is outside the aforementioned thresholds. MyISAM has fast imports and exports of backups and has mysqlhotcopy for faster dumps.

CREATE TABLE t ( i INT ) ENGINE = MYISAM;
CREATE TABLE t ( i INT, j INT, PRIMARY KEY ( i ), INDEX ( j ) ) ENGINE = MYISAM;

A thousand times faster full-text index. MyISAM handles indexes by storing the record byte offset with each index row. The result of this is that all index lookups happen at the same speed and that UPDATEs and INSERTs will not always result in an index update as the byte offset doesn’t always change. The overhead when adding several indexes is also kept at a minimum as each index just need to keep track of the byte offset. INSERT DELAYED will also delay updating indexes until other queries have finished. MyISAM also supports full-text index on VARCHAR, TEXT, etc where each word is stored in a B-tree. Full-text index does consume a lot of space, but a query such as this:

SELECT * FROM articles WHERE body = "% database %";

might run thousands of times faster when a full-text index is added and the query is re-written as:

SELECT * FROM articles (body) MATCH AGAINST ('database');

Please note that MATCH AGAINST won’t display partial word-hits, e.g. “base” won’t locate “database”.

http://dev.mysql.com/doc/refman/5.1/en/myisam-storage-engine.html

InnoDB

The InnoDB engine is an overall good engine that supports some more advanced features that you don’t get with MyISAM with row-level locks and transactions being the most important ones. The row-level lock means that a table can have multiple read and writes at the same time and because it supports Multi version concurrency control (MVCC) a row can be read while being written to. Where there is little system to the data structure in MyISAM, InnoDB orders it by primary key. The result is that for each INSERT, DELETE or UPDATE, rows need to be moved, row position recalculated and indexes potentially updated. This makes INSERTs into a InnoDB table slower than a MyISAM table, but since data is ordered by primary key, selecting based on it is extremely fast. Where MyISAM first reads primary key index file, finds byte offset for the data row and then reads the database file, InnoDB would do this in one step.

InnoDB handles indexes by storing the primary key clustered with all index records, something that could result in a large overhead. It’s therefore recommended to have a small integer or some other space-efficient data type as primary key and not a wide VARCHAR that might be 10-20 times the size.

CREATE TABLE t ( i INT ) ENGINE = INNODB;
CREATE TABLE t ( i INT, j INT, PRIMARY KEY ( i ), INDEX ( j ) ) ENGINE = INNODB;

http://www.innodb.com/innodb

Memory

A MEMORY table is in it self persistent, but its data is not, as it’s stored in memory. This makes it very fast, but prone to data loss. Obvious usage for a MEMORY table is to cache subsets of other tables ( SELECT INTO ) or processed subsets but still retain the power to do SQL queries for even more limited subsets with increased speed. Compared to Memcached this is in some cases far superior in simplicity and probably also in speed. A MEMORY table can of course also be used to store a serialized object, but unless the power of a conditional query is needed, Memcached is probably better for the job. The size of all MEMORY tables are governed by max_heap_table_size with a default value of 16 MB. Worth mentioning is that memory is not recovered when deleting a row in a MEMORY table, but will be reused when inserting a new row. To recover the unused memory do a

ALTER TABLE TABLE ENGINE=MEMORY

to rebuild the table. The default index type is hash, but BTREE is also supported. Using hash is several times faster, but allows identical hashes on identical data that can lead to different kind of issues. These issues are nonexistent with BTREE.

CREATE TABLE t ( i INT ) ENGINE = MEMORY;
CREATE TABLE t ( i INT, INDEX ( i ) ) ENGINE = MEMORY;
CREATE TABLE t ( i INT, INDEX USING BTREE ( i ) ) ENGINE = MEMORY;

http://dev.mysql.com/doc/refman/5.1/en/memory-storage-engine.html

Summary

In conclusion, data in a MEMORY table isn’t persistent but works well for temporary data such as serialized objects or statistics counters and is great for storing calculated subsets of large tables and then do heavy subset calculations on that subset again.

If you don’t need transactions, MyISAM is good for any short table because of the high SELECT/INSERT speed and when the dataset is kept trim the table-lock won’t be a problem. Another case where MyISAM performs well is when a table is either very read- or write-oriented. Multiple SELECTS can query a table without blocking each other and the raw speed of MyISAMs writes results in good performace even with table-lock for each operation.

If you need transactions, InnoDB is one of the better alternatives for MySQL at the moment but is also a good choice for active tables where table-locks will kill your performance.

We’ve had problems with large statistical data in our product NettTV, in combination with MyISAM. Large selects would block the inserts for minutes, at worst. During this period, the tables were effectively constipated, and one would be led to believe that no statistics were gathered during this interval. Later on, the statistics would flod in. MyISAM, referring to the earlier discussion on it, just does not fit with this use-pattern. InnoDB with its row-locks and MVCC probably wouldn’t give the same performance on the statistics part and would still use a lot of CPU but wouldn’t constipate the table the same way.

The newer MySQL engines are exciting but most are at the moment too unstable and with too variable performance for anything but testing, but in a five year perspective a few of the mentioned names probably will see its usage.

Some links on MySQL performance, ordered by importance

http://video.google.com/videoplay?docid=2524524540025172110&q=google+engedu
http://www.amazon.com/gp/product/0596101716
http://www.mysqlperformanceblog.com/2007/05/02/uc2007-presentation-and-notes/
http://jayant7k.blogspot.com/2007/07/mysql-query-cache.html
http://www.petefreitag.com/item/430.cfm
http://forge.mysql.com/wiki/MySQL_Tutorials

Some links on the new MySQL-engines

http://www.mysqlperformanceblog.com/2007/08/01/landscape-of-transactional-storage-engines-for-mysql/
http://forge.mysql.com/wiki/Maria_RoadMap_Design
http://dev.mysql.com/tech-resources/articles/falcon-transactional-engine-part1.html

To be continued

More information related to the MySQL engines can be read in later posts. We will attempt to release one each week to come. Stay tuned using rss.

What you’re probably thinking right now, is “who in the world is Aptoma?” I’ll provide a short answer to that reasonably fair question; Aptoma is a small web-development company situated in Oslo, Norway. We’re something like the Arctic Programmers (which just might explain the A and P in APtoma). We’re passionate about web-development, and we’re passionate about solving problems for the new media. We’ve been so for some three years.

What will we be be blogging about? We’re sure to be blogging about web-development, and everything related to it. Other than that, it’s pretty much an open issue as to what we’ll be focusing on. Still, I’m pretty sure we’ll say hallo to mister MVC, including JavaScript, MySQL, PHP, HTML, performance, frameworks, processes and the likes. And, of course, we’ll follow new media topics and trends. I’m quite sure we’ll be blogging about the specifics on how we put these technologies to good use in creating what we believe to be state of the art web-applications. The only real answer to that question, however, is that we’ll only have to see what comes out of it. We’re quite excited to see for ourselves where this leads us.

Anyways. We’re here, you’re here, we’re a-bloggin’ and we’ll not be going on a brag-blogging-frenzy, like some companies do. We’re on an exploration with this blog and we’re embracing the sharing-for-mutual-benefits model. Our motivation is that our efforts will make us better at what we do. Our main goals for the company, and for ourselves, is to get better at what we do, all the time, and by any means necessary. We figure that the more we put into our blogging-, and thus our sharing-, efforts, the more it’ll be for us to get out of it. Derived from that, you can rest assured that you’ll be presented with the best tips we can offer. We’re just hoping that it’ll be worthwhile your time to follow us, and that you’ll benefit from it and hopefully you’ll let us in on your insights from time to time.

Stay tuned with rss, and we’ll try to treat your limited time respectfully, and we’ll hopefully be leveraging the signal-to-noise-ratio of the web.

We’re starting today with zero subscribers, no readers whatsoever and a lot of catching up to do. Let’s skip the rest of the formalities and get started. Hope to see you around checking up on what we’re doing, how we’re doing it, what we’re fucusing on.