In that case you need to build in slack for learning into the system. You want everyone to have time for sharing, improving and learning continuously, and you want to be able to come home from work with energy to pursue your goals for personal skill development even further.

It’s great fun to be good at what you choose to spend your life doing.

Since the primary raw materials in software product development are skills and intellect, rapid learning is the prime goal and the capital for product development. Fail to invest in raw materials for your business, and you’ll soon run out of stock. Don’t allow for time to learn, rather, require it.

Running a system at 100% yields all sorts of problems.

Symptoms include

• You stop talking to each other. “Sorry, can’t help, have to go do this thing that is  really urgent”
• You start building technical debt. “If I just skip to write that test (it’s really hard to write), then I can save 2 hours today to get to the next thing I have to do.”
• Shallow bug-fixing. “Need to fix this bug real fast, in order to get the other 4 bugs done, too! No time for digging further” Root cause analysis and regression testing get skimped on.
• Useful, although sometimes inconclusive and somewhat directionless, discussions plainly stops happening.
• Stress activates your lizard brain, which numbs the rational being. You are in survival mode.
• Work gets less inspiring and fun, which slows things down. Inspiration is high-octane fuel.
• People get exhausted, and energy-level is low during, and after work hours. You need to “relax” on Facebook at work and in front of the TV at home, instead of doing inspirational learning.

(Picture from Wikimedia commons)

If you have trouble grasping the concept, picture a congested highway. If you remove a small percentage of the cars, you get a magnitude of improvement in throughput and speed. This is a fact.

Did you ever try to squeeze a washing machine 100% full? No clothes get cleaned. Remove 10 or 20% of the clothes, and you have a winner.

Don’t maximize utilization.
Build in time for learning, improving and sharing.

If you, as we have been, find yourself in this mess. There are several ways to get out of it.

1. Say no. If you never say no, you don’t have a strategy — say no to the least important stuff. Saying no is hard, scary and extremely useful.
2. If you work for a consultancy firm, give up. Incentives for maximizing utilization is too high in consultancy companies, as they get paid for “productive hours” only. Quit your job, and go to work in a product development company, or a skilled in-house team. Anything but most consultancy firms, they will cannibalize you.
3. Scale up capacity to meet demand. What can I say, we are hiring.

Quantified

This is no science, but if I would be forced to put some numbers on this, I’d say that:

• Build in approximately 20% learning/sharing time during normal work hours. Yes, that’s one whole day a week. We use it for our monday school, impulsive discussions, reading, friday meeting (our arena for general discussions on processes) and blogging, to mention some activities.
• If you want, you can probably “work” 60 hours a week or more without any danger of burning out, but only as long as that time is spent balanced and having fun, learning and improving/being productive — and only if personal situation allows for it without sacrificing time spent on other important values, like your kids.
• Don’t build overtime into your standard operating procedures. Required overtime doing “productive” work should be the rare exception, not the default.

Find a sustainable pace where you are balancing learning, improving, being productive and all of the time having fun. That’s an important ingredient in the recipe for success.

By the way, did I mention that we’re hiring or that we’re hiring?

The solution we’re currently playing around with is Rhino + Env.js + QUnit.

Rhino is a JavaScript implementation written in Java and you can run it in a terminal, which harmonize with our aim for automation.

Env.js is a DOM implementation written entirely in JavaScript and will simulate the basic functions of a browser.

QUnit is simply the framework for the unit tests.

So far it looks very promising and it looks like we can do much of our testing in this environment. We will post more of our discoveries here later when we have done more testing, so stay tuned or follow me on twitter (@pahen).

In back office applications we can use whatever framework we want. It is used by a small number of people and we have total control over possible dependencies and namespace collisions, so we do not have to worry about breaking other people’s stuff.

When we create code that is going to be included on another web page it is a completely different story. In those cases our goal will be to create code that does not require a framework, so we avoid polluting the requiring site with our dependencies. It might not always be possible to achieve this, but it is a noble goal at least.

We have been using Prototype for most of our work so far. On smaller projects we have started to dabble a bit in JQuery and YUI. The next time we do any major JavaScript development we will explore the options they give us.

The major reasons for moving away from Prototype is the huge size of the library, and numerous changes it makes to basic JavaScript objects. A lot of the changes are good and make for easier code development, but they don’t really feel like proper JavaScript. Instead they rely too much on magical functions and the forced object oriented coding style that Prototype enforces.

We also find that Prototype (and Scriptaculous) are lacking in regards to their option for “plugins”, like calendar- or graph-plugins. Either Scriptaculous have a solution built in, or there are very few good alternatives. Both JQuery and YUI seem to have better communities surrounding them that develop better, and more, plugins that might suit our needs.

The little experience we have with JQuery so far indicates that it is very DOM-centred. This is both a boon and a bane; we loose some of the power we might have gotten used to in Prototype, but it might help us creating purer JavaScript for code in the areas that do not touch on the DOM. This is a goal we have since we want to make all of our JavaScript code testable, and it is much easier to test code that only manipulate data and do not touch the DOM.

Another option in regards to plugins that we are going to look into is ExtJS. The GUI elements that they offer seem to be very powerful, and using pre-built items would save us a lot of time. The pitfall that we saw was that overuse of the items they supply might make the application look too much like a desktop application. Since we are developing for the web we should be careful to not draw too much inspiration from the desktop, they are different media and require different solutions.

We are maintaining a TOC for our JS-discussions here.

Development Tools

Firebug is the most important development tool for us, without comparison. Everyone is familiar with this time-tested tool.

You will rarely need more than Firebug, although there are some valid alternatives now, in 2010. Chrome, Safari and Opera (Dragonfly) have all internal tools for debugging (and more). None of us have used those tools extensively, so we don’t know more than what we have heard from third parties, which is mostly positive. If you need to spot errors in IE, Microsoft Script Debugger is your best hope. IE8 ships with this integrated as default. No-one in our team reported ever making use of the Firebug Light for IE.

When it comes to JavaScript editors, we find that no text editor seems to differ significantly from the rest. Most editors provide color-formatting, some provide API-reference with autocompletion and some editors like vim and TextMate can be configured for auto-validation through JSLint.

Coding Standard

Our coding standard for JavaScript builds on our own internal PHP Coding standard as far as it fits, which in turn builds on the publicly available Zend Framework Coding standard.

We have boiled our standard down to this shortlist.

• Follow the Zend Framework Coding standard
• Use an underscore in front of private objects. (i.e. var _private = 0;) Private objects and methods can be defined as «trusted» with everything else foreign and error-prone. This can make your core more stable and rigid without having to overdo it with data-testing and integrity validation.

It is possible to achieve a private scope in JavaScript, but it comes at a cost in complexity, and we have decided against it. Our decision is based on experience and advice given in “Pro Javascript Design Patterns“.

/**
 * Rationale and short description of ClassName.
 */
var objectName = {
    myFunction: function(whatToDo)
    {
        if (typeof whatToDo === 'string' && whatTodo.length > 0) {
            objectName._doWhatYouGottaDo(whatToDo);
        }
    },
    _doWhatYouGottaDo: function(whatToDo) {
      // As this is a private function, I trust the type and data validity of the argument
      alert(whatToDo);
   }
};

The above is a minimal example, which in no way covers all the rules accurately.

Guidelines

Global scope. Put as little garbage as possible in the global scope. Use well-named objects as namespaces and if needed sub-namespaces. Know the results of using or skipping “var” when creating objects.

Use closures, with care! Closures are necessary in JavaScript, but as not all languages have them, they will to an inexperienced JS-programmer look too much like Magic(TM), which isn’t too good. Use good judgement when to use them, and  remember good inline commenting to make the magic more transparent.

Don’t use the “new” operator. When setting up prototype inheritance it has to be used, but there are no other cases where it should be used. In most cases it’s not only slow, but the alternatives are often easier to understand, require less code and are more powerfull. Because of this, we have decided to conform on code without new. As examples, new Array and new Object should be written as [] and {}.

Prototype inheritance contains power for object oriented code. We will return with more guidelines when our experience grow. We must admit not to have used this elaborately. A serious liability of PrototypeJS (the framework we used) is that it overrides prototype (the language feature) from JavaScript.

Few anonymous functions. As a rule of thumb, we say: name functions. This has to do with testability and readability of your code. The exception is if you need a “one-liner” to use just once, in which case anonymous functions are OK. You would normally not test these kind of functions.

Lint your code! Lint is a code quality tool, and it will hurt your feelings. But  that pain is just Douglas Crockford stabbing the parts of you responsible for writing bad JavaScript. You end up stronger in the end.

Recommended research material

JavaScript requires effort on your behalf. You need to do the homework if you want to be a powerful and successful JavaScript developer. For starters we recommend these resources:

• Closures : Working with closures and Explaining JavaScript scope and closures.
• Douglas Crockford: “JavaScript – The Good Parts” (video)
• Douglas Crockford: “JavaScript – The Good Parts” (amazon)
  

We have decided to address JavaScript with more care. We’d like to give it the time and focus it deserves. Not to be mistaken, we are probably in the top 5% of businesses going deep into JavaScript on our enterprise software, but we still feel we can go further — especially in the fields of quality assurance and scaling.

Lo and behold, we give you;

Monday School Notes from JavaScript session #1

Table of Content

We have decided to divide the subject of JavaScript into the following sections:

• Development Tools
• Coding standard
• Guidelines
• Frameworks, or the lack thereof (when to use what)
• Testing
• Debugging
• Patterns
• Document Object Model (DOM)
• Productivity tips
• Compatibility issues
• Recommended reading/watching

We’ll be covering these topics in the weeks to come, and we well be returning with links on the TOC. If you will join us in this, please follow this blog on RSS or follow the company or the author on twitter. If you digg it, then digg it.

Caching Todo-items

Reverse Proxy Caching and ESI
We will be creating an internal guideline-document for using Varnish, which we’ll also publish here on the blog. Varnish is a simple, fast and stable technology excellently suited for the job (Reverse Proxy Caching and Edge Side Includes).

Data Caching
We will be re-implementing our data-cache layer in our framework, namely AFWCache. The goal is keep options open for future technologies, while solving today’s needs elegantly (as in: sustainable software development). It will become an abstract class which through use of the adapter pattern will support our two technologies of choice, namely Memcache and APC, which serves as both opcode cache and data cache layer.

Future topics for Aptoma Monday School

We’ll be spending this week considering a new set of challenges to thrive on. We’ll conclude this Friday in which direction we’ll move.

• JavaScript Guidelines
• Upgraded database adapter class for our framework

Suggestions? Hang in there (RSS)

We have recently been blogging our notes on different modi operandi of caching. To sum it up, these were: reverse proxy caching, application caching (view caching, subview caching and object/data-caching), opcode caching, client caching and query caching.

We have spent the last two weeks discussing which technologies fits our needs best. Our product installations handle high traffic (i.e. millions of views a day) and we also have products that produce a lot of data that is heavy to compute (i.e. large data sets computed from complex database queries). Thus we have to scale both vertically and horizontally, so to speak. Our needs of caching is thus quite broad, and we have to shop for a lot of technologies in order to fulfill all our needs. This brings us to our …

List of  7 Approved Caching Technologies

In this post we will provide a few notes on each technology.

1. Varnish.
Used for reverse proxy caching. We are not happy with hosting providers ability to configure and fine tune Varnish, so rehearsing your skills in this is certainly not a waste of time.

2. APC Opcode Cache.
Use it. Always. We cannot find examples for when not to. It will speed up execution with no known disadvantages. APC seems to perform better than both competing alternatives (xcache and zend accellerator) under all circumstances.

3. APC for data caching.
You can use APC for data caching, as well. It will outperform memcache by a factor of 10 to 50(!). As opposed to memcache, it does not have distributed access. We have no idea what happens when you exhaust the assigned RAM for data caching. We will have to find out, won’t we?  (Yes we will)

4. Memcache for data caching.
Memcache is still one of the most stable and high performing technologies for its use. It is nevertheless annoying that its performance is significantly slowed down by the fact that it runs on top of TCP/IP (which is also an advantage when it comes to flexibility), even in those cases where only local access is necessary. It seems to us that memcache is run on the same server as the application in more than 90% of all cases.

5. Varnish ESI.
As the name implies, this is a feature of Varnish. ESI is very interesting as it implements subview caching without having to implement it at application level (slower). Implementing subview caching on reverse proxy level can speed things up significantly given the right circumstances. A disadvantage of Varnish ESI is that it introduces more complexity to your source code, as you’ll have to write “Varnish markup” in order to have Varnish do edge side includes for you (<esi:include>). Not a big deal, but definitively a declared enemy of the simplicity ninja. The benefits might make it worthwhile, nevertheless.

6. MySQL for data caching.
This is basically “use MySQL the way you would do memcache”. MySQL for data caching performs quite well, but not really astonishing (memcache is about twice as fast in our tests). MySQL is a convenient technology, as we already require it on all our products, it is also one of the more available technologies regardless of hosting partner (and we have quite a few of them). Another advantage of using database technologies for this purpose is that more sophisticated queries can be applied for purging and invalidating data, than can you in the simpler (but faster) key value databases (memcache et.al).

7. SQLite for data caching.
Same use case as with MySQL above, and with the same pros and cons. It does perform a little better than MySQL. We have decided to look more into this one. Another advantage over memcache is that SQLite and MySQL become persistent caches, (whereas memcache is volatile, i.e. will be blank after a restart).

List of Discarded Technologies

APC as data cache and Varnish ESI are the only technologies which we have not exhaustively used in production for years. Nevertheless we will seek to improve our way of using all these technologies in the time to come, and we will be looking for how to implement support for these in our framework (Aptoma Framework, AFW). To show that we did not just stick to our guns on this one, we present this short list of alternative technologies which we also explored during our tests.

• Tokyo Tyrant (reason to discard : somewhere in between MySQL and Memcache in performance, and brings no new advantages to the blend)
• Nginx (not really a caching technology, merely a faster web-server in special circumstances)
• Lighttpd (Not a caching technology, but can provide faster delivery of static content than Apache can deliver. On dynamic content it does not perform better than a properly stripped and fine tuned Apache.)
• DBA (Does not perform as well as APC in our tests).

We have more discussions to come!

On our wish-list (todo-list) for caching and performance discussions is as follows.

Better benchmarking

• Siege
• Jmeter
• Apache Benchmark (ab)
• Httperf

Caching techniques

1. Pre-loading cache (warming)
2. Event triggered cache invalidation (cache on update)
3. Stale cache (set flag but don’t purge, combined with a grace time in reverse proxy)
4. Better caching on logged in users (Varnish ESI use case)
5. Setting proper headers (for improved client caching and more)
   

Notes From some of our Benchmark Tests

Tokyo Tyrant

Tokyo Tyrant is a memcache-like layer on top of Toky0 Cabinet, which is a fast key-value database. (see : http://sameerparwani.com/posts/tokyo-cabinet-and-tokyo-tyrant) Installation was easy, everything required was available at http://1978th.net/. As a  PHP-wrapper we used http://mamasam.indefero.net/p/tyrant/downloads/2/

We tested Tokyo Tyrant with default settings (defaults matter). The same goes for the comparisons, Memcache and MySQL (query cache is off). The time given bellow is the time it took to set and get 1000 variables off 1 kB.

TokyoTyrant

1. put: 0.102526473999
2. get: 0.121464586258

TokyoTyrant disk hash

1. put: 0.108086037636
2. get: 0.123809480667

TokyoTyrant disk B-Tree

1. put: 0.111338186264
2. get: 0.129682970047

Memcache

1. put: 0.0864425897598
2. get: 0.0702331066132

MySQL without Query Cache

1. put: 0.112287640572
2. get: 0.164221072197

Tokyo Tyrant is slower than Memcache. Tokyo Cabinet can probably outperform memcached if accessed directly, but no PHP-bindings for this purpose were available for our tests.

What is exciting is that Tokyo Tyrant is faster than MySQL for persistent data caching. Tokyo Tyrant is also supposed to have some other features which we have currently did not have time to test. Please share any experiences you might have with Tokyo Tyrant.

MySql, SQLite, DBA, Memcached and APC tests

The test : Write, then read and add to an array 100 000 MD5-hashes. For the relational databases, the insert is done with a multi-row-insert or a  transaction.

MySQL

• Create : 2038.615942 ms
• Read : 13782.1378708 ms

SQLite

• Create : 2084.01703835 ms
• Read : 4064.75901604 ms

DBA (NOTE! Only with 10 000 elements this time)

• Create : 10192.3089027
• Read : 10065.6449795

Memcached

• Create : 3493.89410019
• Read : 3219.08593178

APC

• Create : 25127.6450157 ms
• Read : 172.363996506 ms

MySQL will insert a lot of rows at the same speed as SQLite. W/O query cache, it will be outperformed by a factor of three by SQLite.

SQLite supports :memory: instead of files, which can speed it up as a relational cache, but it is no longer persistent between restarts. MySQLs MEMORY-engine is more of the same.

DBA is a copy-on-write which means that every write will increase its file-size. This makes all operations slower the more writes you have to do. Performance loss was at times huge, but it can be fixed by issuing an optimize-command. DBA writes slowly, (50x slower than the relational databases, and much slower than the other hash-buckets). Reading from a clean file will make it perform somewhere in between the hash-buckets (memcache, APC) and the relational databases

Memcached is a little slower to write to than the relational databases, due to the databases doing all writes in a single command. Read is about the same as with SQLite and only twice as fast as MySql. (Bear in mind that this is a best case scenario for the relational databases)

APC is 5-10x slower than Memcached in writes, but 10-20x faster to read from in this test.

Test conclusions

DBA has few advantages over the relational databases. APC can replace Memcache in some of the areas in which we use memcache today (data/object-caching).

To make things perfectly clear : a cache is a temporary storage area where frequently accessed data can be stored for rapid access.

1. Page caching Reverse proxy caching

Caching the entire page is typically done using Varnish (see the technology references at the end of this article). Our experience with Varnish is that it is stable, and very high performance. However, people who really know how to configure it properly does not come by the dozen. As much as we could hope that all our hosting providers see it as their job to get to know this kind of blazing technology intimately, we can not rely on it, and thus we will have to leverage our own competency further in this area. A weak point of page caching through Varnish, is that once you set a client cookie, Varnish is forced to send traffic straight through (i.e. no caching whatsoever) or dispose of the cookie all together (i.e. break application functionality). Local storage (the “new cookie”, as discussed last week) will not create any such problems for Varnish.

2. View / subview caching (Application Caching)

In MVC-terms, the view is the rendered HTML from the application. Caching the full view would be like the “page caching” from above, only in this case done within the application itself. This would overlap with Varnish strategy above. But even though this version would solve the cookie-problem depicted above, it is normally no longer used this way after Varnish came along.

Subview caching, however, can still be of good use. A subview is simply a part of a page. This is useful when you e.g. have Varnish set for 1 minute caching, but you know that parts of the page could easily be cached for an hour without becoming obsolete. Examples include a seldom changing navigational menu, a generated thumbnail image or page headers and footers. To do subview caching will involve putting your subview data into memcached, file cache or you can achieve something similar using Varnish Edge Side Includes (ESI)

3. Data caching (Application Caching)

Data caching is typically done in your “model-layer” (again with the MVC). A class-function in your business logic will check memcached if there are cached versions of the data about to be fetched from database before proceeding with the SQL-statement if no such data can be found in cache. Afterward it will update the cache with fresh data. This allows a very fine grained control on cache times on different data. Data which is heavy on the ol’ computer and can be cached for a very long time, is candidate even for being serialized and put into a database for easy recovery upon reboots etc. Involving a database for handling large data sets is also provides you with more sophisticated access, update or delete schemes beyond what the neanderthal key/value database in memcached can offer.

We rely heavily on this strategy in our products, and the strategy is typically a good one for creating robust API’s. Also, we use it in our in-house framework (Aptoma FrameWork, AFW) for the Autoloader (automatic discovery and loading of classes) where file-paths are stored in memcached for rapid access upon subsequent requests. Used in combination with APC, this provides us with a blazing fast framework technology.

Let’s take an example

An Event object that handles data in a roster scheduler is stored in the “events” database table. To load an event, we have to fetch a lot of extra information relating to it: Employee, department, workplace, conflicts with interfering events, and so on. All this will be very time consuming. Thus, we should store all data to cache for re-use!

public function loadEvent($id)
{
 // Try to fetch the event from Memcache data storage
 $event = Cache::getObject($id, 'Event');
 if (!empty($event))
 {
  return $event;
 }
 
 // If the Event does not exists in Memcached,
 // we do the conventional, time consuming processing here.
 
 // Finally, we do cache the Event for the posterity
 Cache::setObject($event, 'Event');
 return $event;
}

Usually we use cron-jobs to pre-load these data as well.

A main challenge for this kind of caching-scheme is to prevent data from becoming corrupt. If an interfering event changes, and we fail to validate the cache from the example above, we would be left with bad data in our cache. Keep this in mind when designing your data caching.

4. Local Client caching

Local Client caching is something as simple as the client itself keeping a version of some data, not asking for a new version from the server. This can be cookies, a CSS-file or any other form of static content (javascript or images).

There are two strategies for client caching :
a) force very, very long cache times on the client, and change file names whenever you need the contents of a CSS-file for instance or
b) keep cache times low to ensure that the clients asks for an updated version from time to time.

Strategy a) adds complexity to the application level, as you will have to handle file names changing, and changing file names might break external dependencies on the file. However it will ensure that changes to static content is displayed at the clients immediately. Yahoo recommend this strategy. Strategy b) will delay updates of static content until the cache at the client has expired, but will keep things simpler for the programmer and more stable for any external dependencies on the file names.

5. Query Cache

SQL-query results are cached in MySQL internally. Configuration at server level can be OFF / ON / DEMAND. An attempted cache lookup with no hits, gives a 20% loss of performance. DEMAND requires use of SELECT /*SQL CACHE*/, to force caching. Putting the keyword in comments, will ensure SQL-compatibility with other engines than MySQL.

We recommend using the ON or DEMAND setting on your installation. If you never touched this setting, it is probably set to ON.

If you need to squeeze out every bit of performance available from your application, you should switch to the DEMAND-setting, and review all your SQL-statements and judiciously use the SQL CACHE-trigger. If you have the time and skill to do this, you will benefit from it. You would want to use it on tables with a high percentage of read over write requests. Using SQL-CACHE on tables with a lot of writes will only hurt your overall performance. (Lars has more to say on the topic of Query Cache)

Referred technologies

• Varnish – high-performance HTTP accelerator. (see also Varnish ESI).
• Memcache – high-performance, distributed memory object caching system.
• APC – caching and optimizing PHP intermediate code.
• AFW – our in-house framework which is tightly coupled and integrated to all of the above technologies. (due for release New BSD License in 2010)

With Query Cache, the result set from SELECT queries are cached with the query command itself as key. Meaning that if a SELECT query initially executes in ten seconds, consecutive times the _exact_ (including whitespace, case and more. The query must be an identical string.) same query is executed, the server gets an hit in Query Cache and returns the cached data in milliseconds, normally giving performance boosts between a few and many orders of magnitude. QC is a global server setting and has three modes; ON, OFF and DEMAND. Setting QC to OFF disables it, with it to ON makes the server look for a cached result for all SELECT queries while setting it to DEMAND will have the server only do a QC lookup if the query has SQL_CACHE after the SELECT command. It’s recommended to put this MySQL-only extension in a comment to keep with compatibility with SQL standards. (SELECT /*SQL_CACHE*/ * FROM brille) A QC miss will add up to about 20% overhead compared to the same query executed directly without QC lookup. QC is guaranteed to deliver fresh data as all QC on a table is invalidated when a INSERT, DELETE or UPDATE command is executed on it. This means that for a table with frequent writes, QC on reads will probably decrease performance. On tables with a high number of both reads and writes, setting QC to DEMAND and not use it on queries hitting that table might be a good idea. When actively using query cache on tables with dates it’s also important to write reusable and cacheable queries. Any use of NOW() or equivalent non-deterministic functions will render query cache unused even within the same second, so date and time should be calculated outside of MySQL. When doing so, first analyze the need for an exact query. If you can get away with < 1 minute accuracy, don’t add date( ‘Y-m-d H:i:s’ ) but date( ‘Y-m-d H:i:00′ ) so any identical queries within the same minute will get the cached result. Subselects will not use QC. On high performance applications, QC should be kept in mind when designing the database and queries. Below are two examples where performance characteristics could be radically different:

First table structure:

CREATE TABLE `users` (
`id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`username` varchar(16) collate utf8_danish_ci NOT NULL,
`password` text collate utf8_danish_ci NOT NULL,
`salt` text collate utf8_danish_ci NOT NULL,
`last_time_visited` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY  (`id`),
UNIQUE KEY `username` (`username`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_danish_ci;

Second table structure:

CREATE TABLE `users` (
`id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`username` varchar(16) collate utf8_danish_ci NOT NULL,
`password` text collate utf8_danish_ci NOT NULL,
`salt` text collate utf8_danish_ci NOT NULL,
PRIMARY KEY  (`id`),
UNIQUE KEY `username` (`username`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_danish_ci;
CREATE TABLE `user_visited` (
`id` int(10) UNSIGNED NOT NULL,
`last_time_visited` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY  (`id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_danish_ci;

If the first table was used on a high traffic forum, every time a user loads a page the QC would be invalidated as last_time_visited had to be updated. By denormalization and having two tables where one is often read and the other is often written to, QC on the first table would be kept until a user changed password/username or a new user is registered which is probably less frequent than pageloads.

But the way the SELECT query is created is also important. Lets say you normaly just need the basic information about a user, but every time you visit the users profile, last_time_visited is needed:

First set of queries:
Normal display of user:

SELECT /*SQL_CACHE*/ id, username FROM users WHERE id = 1;

Display of users profile:

SELECT /*SQL_CACHE*/ u.id, u.username, uv.last_time_visited FROM users AS u LEFT JOIN user_visited AS uv ON ( u.id = uv.id ) WHERE u.id = 1;

Second set of queries:
Normal display of user:

SELECT /*SQL_CACHE*/ id, username FROM users WHERE id = 1;

Display of users profile:

SELECT /*SQL_CACHE*/ id, username FROM users WHERE id = 1;
SELECT last_time_visited FROM user_visited WHERE id = 1;

In the first set, two different queries are used to fetch the same information, the username. This means the second query won’t be able to use the QC of the first and more frequent query. Also, the often-read table is joined with the often-written which means INSERT, DELETE and UPDATE queries on _both_ tables will clear cache for this query. In the second set of queries, the same query is used two times, increasing the chance of QC hit. And instead of joining the two tables in one query, a second and separate query is executed on the second table, here without the SQL_CACHE keyword. Two queries increases overhead, but the gain in higher cache hit (The first query can even get a hit in a Memcache-lookup) can easily outweigh this.

AMS is basically an intense hour each Monday @ 0900 hours which we dedicate to educate ourselves through presenting and discussing topics of particular interest. We have a business to run, so the topics has to be of practical use as well as educational value. We will try to identify the sweet-spot of balance between learning something fun and inspiring and creating something tangible of value for our customers.

We have decided to share our lecture notes on the Select *-blog (RSS). The theme of choice for now has been the field of “HTML5“. And we have chosen three topics this week : The <video>-element, the <canvas>-element and local storage.

1. Video

The video-element will start to finally integrate videos fully into the web-page. This means that plugin-technologies such as Adobe Flash or Windows Media Player will no longer be required for video. (yes!)

<video width=”854″ height=”480″ autobuffer controls>
<source src=”big_buck_bunny_480p_stereo.ogx” type=”video/ogg”>
<!– Proceed to define any available fallback formats –>
<source src=”big_buck_bunny_480p_h264.mov”>
Your browser does not support the <code>video</code> element.
</video>

The video-element will now work as a normal element, providing you with the full power of CSS: You can set the css-attribute opacity to 0.5, and it does what you think it will do. If you have been working with any of the plugin-technologies, you’ll know that this is a giant leap forward for video.

The browser can render its browser native controls (play/stop/pause/etc.) or be controlled by JavaScript and make use of event listeners and normal DOM-function to make it “richer” than what is possible with plugins. This basically means that you can design your player using HTML and JavaScript.

See it in action!

Youtube has made an example of use of HTML5/video-element. In this example even animated thumbnails are video-elements displaying how low the resource cost ist. If you did this with Adobe Flash would probably require moarhardware.com to keep all the elements on one page together like that.

A few more details for the very interested reader :

• <video> might just allow us to do more hardware acceleration and the client computer, and hence have a much smaller toll on system resources than today’s solution.
• Safari supports H.264/AAC, Chrome supports H.264/AAC and OGG/Theora/Vorbis while Firefox and Opera just support the latter.
• <video> is supported in Safari, Firefox 3.5, Chrome and a special build of Opera, so we’re not quite there, yet. But we’re getting closer!

2. Canvas

Canvas is another exciting technology which is finally mature for use in most modern web-browsers. Canvas let’s you do pixel by pixel manipulations on images.

Canvas will be a nice replacement for the most server-side generated images as well as for some Flash graphics. Due to the fact that canvas is rendered by the browser and not as a plugin it can be handled as (almost) as all the other HTML elements .

We’ll let this example speak for itself : Remove redeye.

You define a canvas like this
<canvas id=”myCanvas” width=’172′ height=’172′></canvas>

And you will run commands on it like this, to do the funky stuff (presented without any particular context)
var canvas = document.getElementById(“myCanvas”);
var context = canvas.getContext(“2d”); // 3d not yet supported
context.rotate (secondsAngle);
context.drawImage (sechand, -4, -52, 8, 57);
// Rendring text:
context.font         = ‘italic 50px sans-serif’;
context.textBaseline = ‘top’;
context.fillText  (‘Hello world!’, 0, 0);
context.font         = ‘bold 30px sans-serif’;
context.strokeText(‘Hello world!’, 0, 50);

If you want to dig deeper, try these:

• http://developer.apple.com/mac/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Tasks/Canvas.html
• http://virtuelvis.com/archives/2005/12/canvas-image-manipulation
• http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html

3. Local storage

Local storage is what you’d think it is. It might be seen as a sophistication of the good ol’ cookie. It solves a couple of it’s problems, as well. a) You can store more data in local storage. b) You can use the same local storage between browser tabs and windows c) It will be cache-safe, which means that it will not break all your front-end caches (such as Varnish) if used. This is a major improvement for anyone creating high-traffic sites.

The most important functions are

• localStorage.setItem(key, value)
• localStorage.getItem(key)
• localStorage.removeItem(key)
• localStorage.clear() // Use with caution, it will clear all storage.

Local storage will fire events when updated, which means you can monitor the state of the local storage in JavaScript and act upon it with ease.

Future topics for AMS

We maintain this wish-list of topics. We provide no guarantees that we’ll follow the list, but these are the suggested topics as of today.

• HTML5
  • Canvas / SVG
  • Video / Audio
  • Local Storage
  • Web-forms
  • Semantics
  • Worker threads
  • Offline work
• Caching strategies (Memcache, Varnish, Varnish ESI, etc)
• JavaScript development guidelines (JSUnit, JS Coding standard, etc)

We’d love to hear any suggestion you might have for topics in the comments below.