NOTE: This post has been improved at The definitive heatmap
There is not much documentation about creating heat maps. I haven’t been able to find an open source solution that, giving the coordinates, creates a heat map like those shown in Etre blog
In the last post (part 1, part 2) we got a list of click positions in a web page, but the result is easy to improve.

We have the tools. Using a bit of scripting and ImageMagick, it shouldn’t be difficult to make a useful heat map.

The steps

1. Normalizing the data
2. Plotting the dots
3. Capturing the screen
4. Making the overlay

Normalizing the data

Lets assume that the data comes in the following format:

x332y288 http://corunet.com/ 192.168.0.10
x399y288 http://corunet.com/ 192.168.0.10
x489y294 http://corunet.com/ 192.168.0.10
x655y346 http://corunet.com/ 192.168.0.10
x709y351 http://corunet.com/ 192.168.0.10
x350y348 http://corunet.com/ 192.168.0.10
x384y305 http://corunet.com/ 192.168.0.10
…

We need to pick the coordinates and find the boundary values (maximum) and the maximum number of clicks in the same point. This way, we will use that number as the maximum intensity in our heat map. The point most reclicked will have an intensity of 255 and the dots with zero clicks will have zero value. This way, we will probably exceed the maximum value for dots close to each other, but we’ll try to solve it later..

A better way to do it would be to define a matrix as big as the maximum values, add the coordinates and normalyze it, making the biggest value 255 and the smallest one zero, but that will have to wait, at least, until the second version.

The first part of the code (again, in perl) will be like this, marking split lines with #//:


#!/usr/bin/perl
use strict;
my $imgWidth=64;
open (LOG,"log.txt") || die;
my @log=<LOG>;
close (LOG);
my (@x,@y);
my ($xMax,$yMax,$repetitionsMax,$i,$j);
my %repetitions;
foreach my $line(@log){
(my $coords,undef,undef)=split("\t",$line);
$repetitions{$coords}++;
$coords=~s/x//;
($x[$i],$y[$i])=split("y",$coords);
$i++;
}
while ((my $key, my $value)=each(%repetitions)){
$repetitionsMax=$value if $repetitionsMax<$value;
}
for ($j=0;$j<$i;$j++){
$xMax=$x[$j] if $xMax<$x[$j];
$yMax=$y[$j] if $yMax<$y[$j];
}
print $repetitionsMax,"\n$xMax;$yMax\n";


Now, we have three interesting values: xMax, yMax and repetitionsMax. The first two are going to be the dimensions of the canvas, and the third one, our maximum value. For me, those are 848 pixels, 2641 pixels and 3 repetitions. So, lets create the canvas:


my $xCanvas=$xMax+int($imgWidth/2);
my $yCanvas=$yMax+int($imgWidth/2);
my $createcanvas="convert -size ". $xCanvas.#//
"x".$yCanvas." pattern:gray100 empty.png";
system ($createcanvas);


Plotting the dots

We are going to compound our recently created canvas with an existing image for every line in the log. The image, bolilla.png, is a PNG file with transparency that allows us to paint a region instead of a single dot for each click. As the maximum opacity value is 1 (the center of the image is completely opaque) we will use a simple formula to determine the percent of overlay we are going to use: Percent=100/repetitionsMax:


my $createNormalizedSpot="convert bolilla.png -fill #//
white -colorize ".int(100/$repetitionsMax)."% bol.png";
system ($createNormalizedSpot);


So, for each log line, I’ll compound the image again over the canvas:


for ($j=0;$j<$i;$j++){
my ($x,$y);
$x=$x[$j]-32;
$y=$y[$j]-32;
system ("composite -compose multiply -geometry #//
+$x+$y bol.png empty.png empty.png\n");
print "used $j of $i clicks\n";
}


And I got something like this:

Making the final image

We need to remap empty.png to a false color colormap to get the typical heat map appearance. The first thing that come to my mind was to use photoshop, but it’s much more interesting to do it in Imagemagick too. Version 6 is needed to apply colors from one image to another, but once I discovered that, worked like a charm. First of all, we need to invert the image we got:


convert empty.png -negate full.png


And then, replace the colors with


convert full.png colors.png -fx "v.p{0,u*v.h}" final.png


Where colors.png is a gradient image i made first. You can download and use it as you want.

Capturing the screen

The cheapest and easiest way to capture the original web screen is by using Firefox’ Screen Grab. It allows us to capture the complete web page, without having to capture it screen by screen and stitching by hand. What we get is something like this:

I saved the result as blog.png

Compounding the final image

Now, all we have to do is to compound the final image with the screen capture, again from the command line:


composite -blend 40% final.png blog.png heatmap.png


The result

What we got is a heat map of the data in the log. It’s a bit more visual than the last version, and easier to understand when we have lots of dots.

The code

You can download the perl script that does all the work. You will need a working instance of perl (5.6 or better) and ImageMagick V6. Both of them should come as standard in moder linux distributions, or you can install each in a windows system from Activestate (perl) and ImageMagick.org (ImageMagick).

Download code. tgz file, 258KB.

Drop me a line if you use it!

david@corunet.com