How to make heat maps

NOTE: This post has been improved at The definitive heatmap
There is not much documentation about creating heat maps. I haven’t been able to find an open source solution that, giving the coordinates, creates a heat map like those shown in Etre blog
In the last post (part 1, part 2) we got a list of click positions in a web page, but the result is easy to improve.

We have the tools. Using a bit of scripting and ImageMagick, it shouldn’t be difficult to make a useful heat map.

The steps

  1. Normalizing the data
  2. Plotting the dots
  3. Capturing the screen
  4. Making the overlay

Normalizing the data

Lets assume that the data comes in the following format:


We need to pick the coordinates and find the boundary values (maximum) and the maximum number of clicks in the same point. This way, we will use that number as the maximum intensity in our heat map. The point most reclicked will have an intensity of 255 and the dots with zero clicks will have zero value. This way, we will probably exceed the maximum value for dots close to each other, but we’ll try to solve it later..

A better way to do it would be to define a matrix as big as the maximum values, add the coordinates and normalyze it, making the biggest value 255 and the smallest one zero, but that will have to wait, at least, until the second version.

The first part of the code (again, in perl) will be like this, marking split lines with #//:

use strict;
my $imgWidth=64;
open (LOG,"log.txt") || die;
my @log=<LOG>;
close (LOG);
my (@x,@y);
my ($xMax,$yMax,$repetitionsMax,$i,$j);
my %repetitions;
foreach my $line(@log){
(my $coords,undef,undef)=split("\t",$line);
while ((my $key, my $value)=each(%repetitions)){
$repetitionsMax=$value if $repetitionsMax<$value;
for ($j=0;$j<$i;$j++){
$xMax=$x[$j] if $xMax<$x[$j];
$yMax=$y[$j] if $yMax<$y[$j];
print $repetitionsMax,"\n$xMax;$yMax\n";

Now, we have three interesting values: xMax, yMax and repetitionsMax. The first two are going to be the dimensions of the canvas, and the third one, our maximum value. For me, those are 848 pixels, 2641 pixels and 3 repetitions. So, lets create the canvas:

my $xCanvas=$xMax+int($imgWidth/2);
my $yCanvas=$yMax+int($imgWidth/2);
my $createcanvas="convert -size ". $xCanvas.#//
"x".$yCanvas." pattern:gray100 empty.png";
system ($createcanvas);

Plotting the dots

We are going to compound our recently created canvas with an existing image for every line in the log. The image, bolilla.png, is a PNG file with transparency that allows us to paint a region instead of a single dot for each click. As the maximum opacity value is 1 (the center of the image is completely opaque) we will use a simple formula to determine the percent of overlay we are going to use: Percent=100/repetitionsMax:

my $createNormalizedSpot="convert bolilla.png -fill #//
white -colorize
".int(100/$repetitionsMax)."% bol.png";
system ($createNormalizedSpot);

So, for each log line, I’ll compound the image again over the canvas:

for ($j=0;$j<$i;$j++){
my ($x,$y);
system ("composite -compose multiply -geometry #//
+$x+$y bol.png empty.png empty.png\n
print "used $j of $i clicks\n";

And I got something like this:

Making the final image

We need to remap empty.png to a false color colormap to get the typical heat map appearance. The first thing that come to my mind was to use photoshop, but it’s much more interesting to do it in Imagemagick too. Version 6 is needed to apply colors from one image to another, but once I discovered that, worked like a charm. First of all, we need to invert the image we got:

convert empty.png -negate full.png

And then, replace the colors with

convert full.png colors.png -fx "v.p{0,u*v.h}" final.png

Final map image

Where colors.png is a gradient image i made first. You can download and use it as you want.

Capturing the screen

The cheapest and easiest way to capture the original web screen is by using Firefox’ Screen Grab. It allows us to capture the complete web page, without having to capture it screen by screen and stitching by hand. What we get is something like this:

screen capture

I saved the result as blog.png

Compounding the final image

Now, all we have to do is to compound the final image with the screen capture, again from the command line:

composite -blend 40% final.png blog.png heatmap.png

The result

What we got is a heat map of the data in the log. It’s a bit more visual than the last version, and easier to understand when we have lots of dots.

Final heatmap

The code

You can download the perl script that does all the work. You will need a working instance of perl (5.6 or better) and ImageMagick V6. Both of them should come as standard in moder linux distributions, or you can install each in a windows system from Activestate (perl) and (ImageMagick).

Download code. tgz file, 258KB.

Drop me a line if you use it!

4 respuestas a “How to make heat maps”

  1. this is very nice, you are a great programmer with a good sense for visualisation\r\nthe only problem i see is the need for extra manual work for each site – grabbing the website, and i doubt it can be done by some easy script, because it would need to somehow emulate browser.\r\nThat\’s why i really like the original idea with two or more size/opacity/color/z-index images, because it do not need any programming and if the other problems mentioned before with flexible widths etc. are solved, than it is an ideal sollution, it can look like this , with better image as you use even better, close to your sollution, but with no need to make one image.
    result could by parsed by some ajax xml parser, where you set the div of the whole page (container in your case), and do document write div id position absolute left 0 top 0 width:100% height:100% / img blue z-index 1 position absolute left 10 top 20 img right z-index 10 position absolute left 10 top 20

  2. Thanks, noname.
    Your idea is perfectly viable with this system. You would only need to change the last part of the image generation. Instead of composing both images (last step) just create a semi-transparent png and overlay it using last article’s technique

    This way you wouldn’t need to manually capture the screen. But that wouldn’t be difficult to automate, anyway. Stay tuned!

Comentarios cerrados.