The definitive heatmap

David Pardo — Wed, 16 Aug 2006 18:39:21 +0000

After the interest shown about the clickmaps / heatmaps articles, I’ve decided to gather all the information into an easy to use system. What we are going to make is a complete solution that allows collecting, analyzing and showing the click information our users give us. Now, it works in web pages not center aligned and is quite a bit more robust. Read on…

What?

If you are a webmaster, you had probably thought about what do users do in your website. Beyond usual statistics, clickmaps allow you to find where your users are clicking. This is quite useful to find areas in needing of change, layouts that don’t work as intended or anchors that aren’t being understood as you would like.

You’re going to be able to find every single click your users make in your website, being over a link or even in blank areas. We are going to do it the following way:

The proccess

We need to divide the full proccess into some manageable steps that use some open source tools. Since I work both in windows and linux systems, I’ll be OS agnostic and use only tools available in most systems, including Mac OSX.
The main steps and the tools they use are the following:

The collecting (javascript and apache)
The processing (ruby and imageMagick)
The showing (javascript)

The collecting

We are going to use a small snippet of unobtrusive javascript to allow the client to tell our server the click positions. Just place this small javascript file at the very end of your template, right before the closing tag:

registerclicks.js

/**
 * @author David Pardo: Corunet
 * Run after loading
 */

var xOffset,yOffset;
var tempX = 0;
var tempY = 0;

//detect browser
var IE = document.all?true:false
if (!IE) {
	document.captureEvents(Event.MOUSEMOVE)
}
//find the position of the first item on screen and store offsets
	//find the first item on screen (after body)
	var firstElement=document.getElementsByTagName('body')[0].childNodes[1];
	//find the offset coordinates
	xOffset=findPosX(firstElement);
	yOffset=findPosY(firstElement);
	if (IE){ // In IE there's a default margin in the page body. If margin's not defined, use defaults
		var marginLeftExplorer  = parseInt(document.getElementsByTagName('body')[0].style.marginLeft);
		var marginTopExplorer   = parseInt(document.getElementsByTagName('body')[0].style.marginTop);
		/*assume default 10px/15px margin in explorer*/
		if (isNaN(marginLeftExplorer)) {marginLeftExplorer=10;}
		if (isNaN(marginTopExplorer)) {marginTopExplorer=15;}
		xOffset=xOffset+marginLeftExplorer;
		yOffset=yOffset+marginTopExplorer;
	}
/*attach a handler to the onmousedown event that calls a function to store the values*/
document.onmousedown = getMouseXY;

/*Functions*/
/*Find positions*/
function findPosX(obj){
	var curleft = 0;
	if (obj.offsetParent){
		while (obj.offsetParent){
			curleft += obj.offsetLeft
			obj = obj.offsetParent;
		}
	}else if (obj.x){
		curleft += obj.x;
	}
	return curleft;
}

function findPosY(obj){
	var curtop = 0;
	if (obj.offsetParent){
		while (obj.offsetParent){
			curtop += obj.offsetTop
			obj = obj.offsetParent;
		}
	}else if (obj.y){
		curtop += obj.y;
	}
	return curtop;
}
function getMouseXY(e) {
	if (IE) {
		tempX = event.clientX + document.body.scrollLeft
		tempY = event.clientY + document.body.scrollTop
	} else {
		tempX = e.pageX
		tempY = e.pageY
	}
	tempX-=xOffset;
	tempY-=yOffset;
	var url='guardacoordenadas.pl?x='+tempX+'&y='+tempY;
	guardar(url);
	return true;
}
function guardar(url){
	var xmlDoc = null ;
	if (typeof window.ActiveXObject != 'undefined' ) {
		xmlDoc = new ActiveXObject('Microsoft.XMLHTTP');
	}else {
		xmlDoc = new XMLHttpRequest();
	}
	xmlDoc.open( 'GET', url, true );
	xmlDoc.send( null );
}

The code adds a onMouseDown handler to the document, executes a function for every click and returns true, since we want the user to follow the normal navigation. Then, when the user clicks any part of the page, a tiny request is going to get sent on the background to our server. The script has to calculate the offsett of the first element inside the tag, because most pages arent aligned to the top-right corner. In liquid layouts the system is not going to work at all

The request is sent via a HttpRequest object that calls a file in the server. In last version, I used a small GCI written in perl to log the request and return an empty document, but since we want to serve so many request, there’s a better method to apply. Using a perl CGI, in a modern server, we get the following results benchmarking with apache bench (100 requests, 10 concurrent ones):

Concurrency Level: 10 Time taken for tests: 6.537187 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 17100 bytes HTML transferred: 0 bytes Requests per second: 15.30 [#/sec] (mean) Time per request: 653.719 [ms] (mean) Time per request: 65.372 [ms] (mean, across ...) Transfer rate: 2.45 [Kbytes/sec] received

Mod_imap

Apache has some modules that work the following way:
You define a handler and what you want to do with it. Some of them are well known, like mod_perl or mod_cgi, but a lesser known one, called mod_imap, does exactly what we want. It’s a module meant to return server-side image maps, but if we use an empty map file, all we get is a 204 status (no data) and a logged transaction. The difference is quite significative. Using Apache Bench with the same configuration, this is what we get:

Concurrency Level: 10 Time taken for tests: 0.106316 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 36464 bytes HTML transferred: 20246 bytes Requests per second: 940.59 [#/sec] (mean) Time per request: 10.632 [ms] (mean) Time per request: 1.063 [ms] (mean, across ...) Transfer rate: 329.21 [Kbytes/sec] received

That’s 950 requests per second vs 15 with the CGI method!!! We are almost a hundred times faster with this approach!The only thing we have to do to use this mod_imap is to touch a little bit the apache configuration file. Do it carefully because it can hurt your entire server. In the relevant section add the following lines:

AddHandler mod_imap .map CustomLog /tmp/clicklog clicklog #or modify according to your system

And define a custom log in the same file adding this:
LogFormat "%q,%{Referer}i" clicklog

This way, everything ending in .map is going to be treated as a server-side map, and since the map is empty, it’s not taking your user anywhere. But it logs it, in file /tmp/clicklog (YMMV).

The log analysis

Since we used a logFormat apache directive to write our log, the format should be easy to parse. The query string is written in the log as it comes, and the full lines should be in the following format:

?x=483&y=32&dx10&dy15,http://demo.html ?x=461&y=177&dx10&dy15,http://demo.html ?x=408&y=40&dx10&dy15,http://demo.html (...)

I decided to write a Ruby script to parse the file and generate the final images, because I hadn’t used ruby before and thought it would be a good way to approach the problem. Last time I had written an structured perl script, but I think that object-oriented is the way to go in this particular situation, since the objects should be well-defined and dividing the program among several coders should be easier too.

Update:Thanks to Jerret, this part has been enhanced using RMagick. Part of the code below can be updated and works some 50 times faster. On top of that, a new sourceforge project has been started at http://sourceforge.net/projects/clickmaps/ under a GPL license. Of course, if you don’t want to install/use RMagick you can still download the original version at the end of this post.

I´ll try to explain the model. It uses five classes:

Conf:: Sets some configuration variables and returns them as a hash. This way, every configuration variable is set in this class and it’s easy to get them later on

conf.rb

class Conf
	def initialize
	@data = {
		'logfile' => 'log.txt',
		'dotimage' => 'bolilla.png',
		'format' => 'png',
		'colorimage' => 'colors.png',
		'opacity' => "0.50",
		'dotwidth' => 64
	}
	raise "log file not found" \
	unless File.exist?(@data['logfile'])
	raise "dot image not found" \
	unless File.exist?(@data['dotimage'])
	raise "color image not found" \
	unless File.exist?(@data['colorimage'])
	end
	attr_reader :data
end

Readparsefile: Reads and parses the file defined as logfile in the conf object. For each log line, it stores it into a click object and append it to an array. There are two methods that return all the URLs in the log file (geturls) and all the information for a single URL as a Log object

readparsefile.rb

class Readparsefile
  def initialize(name)
    @name = name
    @data = Array.new
    lines = IO.readlines(@name).collect { |l| l.chomp }
    for line in lines
        line.gsub!(/\?x\=/,'')
        line.gsub!(/\&y\=/,',')
        line.gsub!(/\//,'_')
        x,y,url = line.split(/,/)
        if (x and y and url)
            @data.push(Click.new(url, x, y))
        else
            $stderr.puts "Warning: Bogus line "<< line
        end
    end
    @urls = Array.new
    @data.each do |line|
        @urls.push(line.url)
    end
    raise "no clicks found" unless lines.length > 0
    @urls.uniq!
  end
  def geturls
      return @urls
  end
  def coordsurl(url)
    @url=url
    xMax=0
    yMax=0
    coords = Array.new
    @data.each do |line|
        coords.push(line)
        xMax=line.x if line.x>xMax
        yMax=line.y if line.y>yMax
    end
    return Log.new(xMax,yMax,coords,@url)
  end
end

Click: Stores the data in each log line, including X, Y and URL. Provides a method (xy) that returns an string like “x100y200″ to compare the exact coordinates, useful to extract the maximum number of times a single click is repeated

click.rb

class Click
    def initialize(url,x,y)
        @url=url
        @x=x
        @y=y
    end
    def url
        return @url
    end
    def x
        return @x.to_i
    end
    def y
        return @y.to_i
    end
    def xy
        return "x"+@x+"y"+@y
    end
end

Log: Stores all the values pertinent to a single URL and gives accesors to them. There’s also a “next” method that returns next click within the same URL

log.rb

class Log
    def initialize (x,y,list,url)
        @line = 0
        @x=x
        @y=y
        @url=url
        @list=list
        @points = Hash.new(0)
        @list.each do |point|
            @points[point.xy] +=1
        end
        @reps = @points.values.max
    end
    attr_reader  , :y, :list, :reps, :url
    def next
        coord = @list[@line]
        @line += 1
        return coord
    end
end

Image: Receives a log object and the conf object. There are three methods to normalize the spot we’re going to use as a click indicator (normalizespot), compose every click as a dot (iterate) and colorize the final image (colorize)

image.rb

class Image
    def initialize(data,conf)
        @data = data
        @name = data.url.gsub!(/\W/,'').gsub!(/\_/,'').to_s
        @conf=conf.data
    end
    def normalizespot
        #divide spot.png intensity by max. position reps (@data.reps)
        intensity = (100-(100/@data.reps).ceil).to_s
        normalize = "convert "<<@conf['dotimage']<<" -fill white -colorize "<

Then, the main program is only eight lines long. It leverages the objects’ methods to be as compact as possible. In fact, the only thing it does is to iterate over each url to create a different image.

conf = Conf.new
file = Readparsefile.new(conf.data['logfile'])
file.geturls.each do |url|
    image = Image.new(file.coordsurl(url),conf)
    image.normalizespot
    image.iterate
    image.colorize
end

Is it better?

You can find another program (this time written in perl) in an older post that does a similar job of making heatmaps. But there has been some modifications that makes this an usable system instead of a proof of concept:

Flexible configuration: Over the harcoded last version, in this one is quite simple to modify the images used in the heatmap generation, or the log name. You only have to modify the Conf definition. It would be so easy to use an external conf file, but doing it this way is quicker for me
Multiple URL support: While last version only let you extract one image, this one makes a heatmap for every URL in your log.
Much faster execution time: Instead of composing the full image everytime, now we create a single ImageMagick sentence to do al the composition for us. That gives us a couple of orders speed advantage. Last version lasted about fifteen minutes for a couple hundred clicks, and now it’s about five seconds. Please note that, for many clicks, the program uses quite a bit of memory. Probably for a production environment it would be neccesary to divide the compose sentence into manageable chuncks, and iterate at the end with them to create the final heatmap.
Manual capture is not needed anymore: Since the last step is to decrement the opacity of the map, we can use a little bit of javascript to overlay the PNG image over the original page. So, the stakeholders can review it without someone manually capturing the screen. This way we don’t need to set an XServer in the production environment
Easier to maintain and extend: The object oriented paradigm doesn’t give us faster code, but much more manageable one. You can extend it as you want

What you get

Now, you’ll have several images. Most of them are OK to delete, but there’s one ending in final.png that’s your heatmap. We’re going to overlay it on top of your web page. That image should be a semi-transparent PNG like this one:

The overlay

This is the final part of the proccess. We already have the overlay image and all we need is a javascript snippet that can be called anytime and that creates a layer on top of your website with the click information. Just like the first step, we’re going to position it over the very first item in the page.
The best way to do that is via a bookmarklet, that is, an small javascript snippet saved as a bookmark. This way, you can have it in your browser and ask for the overlay image when you feel like. The javascript recalculates the offsets of the first element inside the tag and writes the heatmap image on top of it.

overlay.js

/**
 * @author DavidPardo
 */
//find the position of the first item on screen and store offsets
//find the first item on screen (after body)
var firstElement=document.getElementsByTagName('body')[0].childNodes[1];
//find the offset coordinates
xOffset=findPosX(firstElement);
yOffset=findPosY(firstElement);
if (IE){ // In IE there's a default margin in the page body. If margin's not defined, use defaults
	var marginLeftExplorer  = parseInt(document.getElementsByTagName('body')[0].style.marginLeft);
	var marginTopExplorer   = parseInt(document.getElementsByTagName('body')[0].style.marginTop);
	/*assume default 10px/15px margin in explorer*/
	if (isNaN(marginLeftExplorer)) {marginLeftExplorer=10;}
	if (isNaN(marginTopExplorer)) {marginTopExplorer=15;}
	xOffset=xOffset+marginLeftExplorer;
	yOffset=yOffset+marginTopExplorer;
}
// add the image element to the dom, absolutely positioned
var style="z-index:100;position:absolute;left:"+xOffset+"px;top:"+yOffset+"px";
var bodytag=document.getElementsByTagName('body')[0]
var newdiv = document.createElement('div');
var currentpage=location.href;
var escapedpage=currentpage.replace(/\W/g,'');
newdiv.setAttribute('id','heatmap');
newdiv.innerHTML = '';
bodytag.appendChild(newdiv);

The result

We got a beautiful heatmap on top of our web page. We can call the overlay from wherever we want and show it to the project stakeholders. Look at the result:

The code

I made a ready to download package with all the code. It’s released under a MIT license that means that you can do whatever you want with it. Probably in the future it’ll be part of an open source release; if you feel like, start it yourself or contact me for more information.

Download code. Tar.gz file

What else?

The sky is the limit. If you want a hosted service, contact us. Our company can give you bespoke solutions to all your web intelligence needs, being it log analysis, path tracking and so on. If you’re a developer, feel free to use all the code as you wish, and please write me to tell your experiences. Stay tuned!

By the way, there has been a post in remysharp blog explaining how to record the clicks in a different server. Thanks.

Zero budget eye tracking. Clickmaps part 2

David Pardo — Sun, 30 Jul 2006 16:01:08 +0000

In the first part of Zero budget eye tracking, we wrote a small script to keep track of the clicks in a web page. A couple of days after, we already have some information to analyze. Keep on reading to know how…
NOTE: This post has been improved at The definitive heatmap
I added the scripts to our website (www.corunet.com) and used a resolution of 1280×1024 px. After 65 clicks, I have a log like this one:

x332y288 http://corunet.com/ 192.168.0.10 x399y288 http://corunet.com/ 192.168.0.10 x489y294 http://corunet.com/ 192.168.0.10 x655y346 http://corunet.com/ 192.168.0.10 x709y351 http://corunet.com/ 192.168.0.10 x350y348 http://corunet.com/ 192.168.0.10 x384y305 http://corunet.com/ 192.168.0.10 ...

We would need to filter the x and y coordinates to draw some dots over the page. I will use the following code (written in perl):

#!/usr/bin/perl open (DATOS,"log.txt"); my @datos=<DATOS>; close (DATOS); my $i; foreach my $dato(@datos){ if ($dato=~/\.com\/\t/){ my @posicion=split("\t",$dato); $posicion[0]=~s/x//; my ($x,$y)=split("y",$posicion[0]); $i++; print ' style="z-index:'.$i.';position:absolute; left:'.$x.'px;top:'.$y.'px; height:16px;width:16px;">'."\n"; } }

This script writes out an IMG tag for every log line that asked for the main page (ends in .com). This way, we would only need to paste this html code into a copy of the original page. We also will need a small image to mark the spots. I used a 16×16px

The click map

The former script prints out some html code like this one:

... style="z-index:51;position:absolute; left:605px;top:773px; height:16px;width:16px;"> style="z-index:52;position:absolute; left:778px;top:792px; height:16px;width:16px;"> style="z-index:53;position:absolute; left:799px;top:810px; height:16px;width:16px;"> ...

Pasting the code into the page copy, this is what we get:

I used the system posted in webfx to use transparent PNGs in internet explorer. We wouldn’t need it if we had done our research using Opera or Firefox browsers.
Each dot in the above image refers to a single click in our web site. Studying it we would be able to identify areas where the users are having trouble to find the links, and where they think should be a link but isn’t. In a real world analysis it would be better to have some more dots to work on.

Shortcomings

The system’s not able to support resolution changes. We would need some more javascript to refer every click to a known point of the screen. And it would be pretty difficult, if not impossible, to use it in liquid layouts.

In flash movies, we are going to have some trouble because the onclick event is not supported. As you can see in the screen capture above, there is not a single click detected in the top area, where the flash menu is.

Using the same technique, we could be able to store every mouse position in some milliseconds interval to replay our users movements, and there are many other possibilities.

Acknowledgements

I got the inspiration for this post from the “Click Survey” that lets you click randomly in any position of a picture.

Corunet. El Blog » heatmaps