Corunet. El blog

The definitive heatmap

Archivado en English, Usability por david el día 16 de Agosto de 2006

After the interest shown about the clickmaps / heatmaps articles, I’ve decided to gather all the information into an easy to use system. What we are going to make is a complete solution that allows collecting, analyzing and showing the click information our users give us. Now, it works in web pages not center aligned and is quite a bit more robust. Read on…

What?

If you are a webmaster, you had probably thought about what do users do in your website. Beyond usual statistics, clickmaps allow you to find where your users are clicking. This is quite useful to find areas in needing of change, layouts that don’t work as intended or anchors that aren’t being understood as you would like.

You’re going to be able to find every single click your users make in your website, being over a link or even in blank areas. We are going to do it the following way:

The proccess

We need to divide the full proccess into some manageable steps that use some open source tools. Since I work both in windows and linux systems, I’ll be OS agnostic and use only tools available in most systems, including Mac OSX.
The main steps and the tools they use are the following:

The collecting (javascript and apache)
The processing (ruby and imageMagick)
The showing (javascript)

The collecting

We are going to use a small snippet of unobtrusive javascript to allow the client to tell our server the click positions. Just place this small javascript file at the very end of your template, right before the closing <body> tag:

Code: registerclicks.js

/**
 * @author David Pardo: Corunet
 * Run after loading
 */
 
var xOffset,yOffset;
var tempX = 0;
var tempY = 0;
 
//detect browser
var IE = document.all?true:false
if (!IE) {
	document.captureEvents(Event.MOUSEMOVE)
}
//find the position of the first item on screen and store offsets
	//find the first item on screen (after body)
	var firstElement=document.getElementsByTagName('body')[0].childNodes[1];
	//find the offset coordinates
	xOffset=findPosX(firstElement);
	yOffset=findPosY(firstElement);
	if (IE){ // In IE there's a default margin in the page body. If margin's not defined, use defaults
		var marginLeftExplorer  = parseInt(document.getElementsByTagName('body')[0].style.marginLeft);
		var marginTopExplorer   = parseInt(document.getElementsByTagName('body')[0].style.marginTop);
		/*assume default 10px/15px margin in explorer*/
		if (isNaN(marginLeftExplorer)) {marginLeftExplorer=10;}
		if (isNaN(marginTopExplorer)) {marginTopExplorer=15;}
		xOffset=xOffset+marginLeftExplorer;
		yOffset=yOffset+marginTopExplorer;
	}
/*attach a handler to the onmousedown event that calls a function to store the values*/
document.onmousedown = getMouseXY;
 
 
 
/*Functions*/
/*Find positions*/
function findPosX(obj){
	var curleft = 0;
	if (obj.offsetParent){
		while (obj.offsetParent){
			curleft += obj.offsetLeft
			obj = obj.offsetParent;
		}
	}else if (obj.x){
		curleft += obj.x;
	}
	return curleft;
}
 
function findPosY(obj){
	var curtop = 0;
	if (obj.offsetParent){
		while (obj.offsetParent){
			curtop += obj.offsetTop
			obj = obj.offsetParent;
		}
	}else if (obj.y){
		curtop += obj.y;
	}
	return curtop;
}
function getMouseXY(e) {
	if (IE) {
		tempX = event.clientX + document.body.scrollLeft
		tempY = event.clientY + document.body.scrollTop
	} else {
		tempX = e.pageX
		tempY = e.pageY
	}
	tempX-=xOffset;
	tempY-=yOffset;
	var url='guardacoordenadas.pl?x='+tempX+'&y='+tempY;
	guardar(url);
	return true;
}
function guardar(url){
	var xmlDoc = null ;
	if (typeof window.ActiveXObject != 'undefined' ) {
		xmlDoc = new ActiveXObject('Microsoft.XMLHTTP');
	}else {
		xmlDoc = new XMLHttpRequest();
	}
	xmlDoc.open( 'GET', url, true );
	xmlDoc.send( null );
}

The code adds a onMouseDown handler to the document, executes a function for every click and returns true, since we want the user to follow the normal navigation. Then, when the user clicks any part of the page, a tiny request is going to get sent on the background to our server. The script has to calculate the offsett of the first element inside the <body> tag, because most pages arent aligned to the top-right corner. In liquid layouts the system is not going to work at all

The request is sent via a HttpRequest object that calls a file in the server. In last version, I used a small GCI written in perl to log the request and return an empty document, but since we want to serve so many request, there’s a better method to apply. Using a perl CGI, in a modern server, we get the following results benchmarking with apache bench (100 requests, 10 concurrent ones):

Concurrency Level: 10 Time taken for tests: 6.537187 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 17100 bytes HTML transferred: 0 bytes Requests per second: 15.30 [#/sec] (mean) Time per request: 653.719 [ms] (mean) Time per request: 65.372 [ms] (mean, across ...) Transfer rate: 2.45 [Kbytes/sec] received

Mod_imap

Apache has some modules that work the following way:
You define a handler and what you want to do with it. Some of them are well known, like mod_perl or mod_cgi, but a lesser known one, called mod_imap, does exactly what we want. It’s a module meant to return server-side image maps, but if we use an empty map file, all we get is a 204 status (no data) and a logged transaction. The difference is quite significative. Using Apache Bench with the same configuration, this is what we get:

Concurrency Level: 10 Time taken for tests: 0.106316 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 36464 bytes HTML transferred: 20246 bytes Requests per second: 940.59 [#/sec] (mean) Time per request: 10.632 [ms] (mean) Time per request: 1.063 [ms] (mean, across ...) Transfer rate: 329.21 [Kbytes/sec] received

That’s 950 requests per second vs 15 with the CGI method!!! We are almost a hundred times faster with this approach!The only thing we have to do to use this mod_imap is to touch a little bit the apache configuration file. Do it carefully because it can hurt your entire server. In the relevant section add the following lines:

AddHandler mod_imap .map CustomLog /tmp/clicklog clicklog #or modify according to your system

And define a custom log in the same file adding this:
LogFormat "%q,%{Referer}i" clicklog

This way, everything ending in .map is going to be treated as a server-side map, and since the map is empty, it’s not taking your user anywhere. But it logs it, in file /tmp/clicklog (YMMV).

The log analysis

Since we used a logFormat apache directive to write our log, the format should be easy to parse. The query string is written in the log as it comes, and the full lines should be in the following format:

?x=483&y=32&dx10&dy15,http://demo.html ?x=461&y=177&dx10&dy15,http://demo.html ?x=408&y=40&dx10&dy15,http://demo.html (...)

I decided to write a Ruby script to parse the file and generate the final images, because I hadn’t used ruby before and thought it would be a good way to approach the problem. Last time I had written an structured perl script, but I think that object-oriented is the way to go in this particular situation, since the objects should be well-defined and dividing the program among several coders should be easier too.

Update:Thanks to Jerret, this part has been enhanced using RMagick. Part of the code below can be updated and works some 50 times faster. On top of that, a new sourceforge project has been started at http://sourceforge.net/projects/clickmaps/ under a GPL license. Of course, if you don’t want to install/use RMagick you can still download the original version at the end of this post.

I´ll try to explain the model. It uses five classes:

Conf:: Sets some configuration variables and returns them as a hash. This way, every configuration variable is set in this class and it’s easy to get them later on

Code: conf.rb

class Conf
	def initialize
	@data = {
		'logfile' => 'log.txt',
		'dotimage' => 'bolilla.png',
		'format' => 'png',
		'colorimage' => 'colors.png',
		'opacity' => "0.50",
		'dotwidth' => 64
	}
	raise "log file not found" \
	unless File.exist?(@data['logfile'])
	raise "dot image not found" \
	unless File.exist?(@data['dotimage'])
	raise "color image not found" \
	unless File.exist?(@data['colorimage'])
	end
	attr_reader :data
end

Readparsefile: Reads and parses the file defined as logfile in the conf object. For each log line, it stores it into a click object and append it to an array. There are two methods that return all the URLs in the log file (geturls) and all the information for a single URL as a Log object

Code: readparsefile.rb

class Readparsefile
  def initialize(name)
    @name = name
    @data = Array.new
    lines = IO.readlines(@name).collect { |l| l.chomp }
    for line in lines
        line.gsub!(/\?x\=/,'')
        line.gsub!(/\&y\=/,',')
        line.gsub!(/\//,'_')
        x,y,url = line.split(/,/)
        if (x and y and url)
            @data.push(Click.new(url, x, y))
        else
            $stderr.puts "Warning: Bogus line "<< line
        end
    end
    @urls = Array.new
    @data.each do |line|
        @urls.push(line.url)
    end
    raise "no clicks found" unless lines.length > 0
    @urls.uniq!
  end
  def geturls
      return @urls
  end
  def coordsurl(url)
    @url=url
    xMax=0
    yMax=0
    coords = Array.new
    @data.each do |line|
        coords.push(line)
        xMax=line.x if line.x>xMax
        yMax=line.y if line.y>yMax
    end
    return Log.new(xMax,yMax,coords,@url)
  end
end

Click: Stores the data in each log line, including X, Y and URL. Provides a method (xy) that returns an string like “x100y200″ to compare the exact coordinates, useful to extract the maximum number of times a single click is repeated

Code: click.rb

class Click
    def initialize(url,x,y)
        @url=url
        @x=x
        @y=y
    end
    def url
        return @url
    end
    def x
        return @x.to_i
    end
    def y
        return @y.to_i
    end
    def xy
        return "x"+@x+"y"+@y
    end
end

Log: Stores all the values pertinent to a single URL and gives accesors to them. There’s also a “next” method that returns next click within the same URL

Code: log.rb

class Log
    def initialize (x,y,list,url)
        @line = 0
        @x=x
        @y=y
        @url=url
        @list=list
        @points = Hash.new(0)
        @list.each do |point|
            @points[point.xy] +=1
        end
        @reps = @points.values.max
    end
    attr_reader :x, :y, :list, :reps, :url
    def next
        coord = @list[@line]
        @line += 1
        return coord
    end
end

Image: Receives a log object and the conf object. There are three methods to normalize the spot we’re going to use as a click indicator (normalizespot), compose every click as a dot (iterate) and colorize the final image (colorize)

Code: image.rb

class Image
    def initialize(data,conf)
        @data = data
        @name = data.url.gsub!(/\W/,'').gsub!(/\_/,'').to_s
        @conf=conf.data
    end
    def normalizespot
        #divide spot.png intensity by max. position reps (@data.reps)
        intensity = (100-(100/@data.reps).ceil).to_s
        normalize = "convert "<<@conf['dotimage']<<" -fill white -colorize "<<intensity<<"% "<<@name<<".bol.png"
        system(normalize)
    end
    def iterate
        halfwidth=@conf['dotwidth']/2
        compose = "convert -page "<<(@data.x+halfwidth).to_s<<"x"<<(@data.y+halfwidth).to_s<<" pattern:gray100 "
        #iterate spots
        @data.list.each do |dot|
            compose << "-page +"<<((dot.x)-halfwidth).to_s<<"+"<<((dot.y)-halfwidth).to_s<<" "<<@name<<".bol.png "
        end
        compose << "-background white -compose multiply -flatten "<<@name<<".empty.png"
        system(compose)
    end
    def colorize
        #invert image...
        invert = "convert "<<@name<<".empty.png -negate "<<@name<<".full.png"
        system(invert)
        #colorize it...
        colorize = "convert "<<@name<<'.full.png -type TruecolorMatte '<<@conf['colorimage']<<' -fx "v.p{0,u*v.h}" '<<@name<<".colorized.png"
        system(colorize)
        #and apply transparency...
        transparency = "convert "<<@name<<'.colorized.png -channel A -fx "A*'<<@conf['opacity']<<'"  '<<@name<<'.final.'<<@conf['format']
        system(transparency)
    end
end

Then, the main program is only eight lines long. It leverages the objects’ methods to be as compact as possible. In fact, the only thing it does is to iterate over each url to create a different image.

conf = Conf.new
file = Readparsefile.new(conf.data['logfile'])
file.geturls.each do |url|
    image = Image.new(file.coordsurl(url),conf)
    image.normalizespot
    image.iterate
    image.colorize
end

Is it better?

You can find another program (this time written in perl) in an older post that does a similar job of making heatmaps. But there has been some modifications that makes this an usable system instead of a proof of concept:

Flexible configuration: Over the harcoded last version, in this one is quite simple to modify the images used in the heatmap generation, or the log name. You only have to modify the Conf definition. It would be so easy to use an external conf file, but doing it this way is quicker for me
Multiple URL support: While last version only let you extract one image, this one makes a heatmap for every URL in your log.
Much faster execution time: Instead of composing the full image everytime, now we create a single ImageMagick sentence to do al the composition for us. That gives us a couple of orders speed advantage. Last version lasted about fifteen minutes for a couple hundred clicks, and now it’s about five seconds. Please note that, for many clicks, the program uses quite a bit of memory. Probably for a production environment it would be neccesary to divide the compose sentence into manageable chuncks, and iterate at the end with them to create the final heatmap.
Manual capture is not needed anymore: Since the last step is to decrement the opacity of the map, we can use a little bit of javascript to overlay the PNG image over the original page. So, the stakeholders can review it without someone manually capturing the screen. This way we don’t need to set an XServer in the production environment
Easier to maintain and extend: The object oriented paradigm doesn’t give us faster code, but much more manageable one. You can extend it as you want

What you get

Now, you’ll have several images. Most of them are OK to delete, but there’s one ending in final.png that’s your heatmap. We’re going to overlay it on top of your web page. That image should be a semi-transparent PNG like this one:

The overlay

This is the final part of the proccess. We already have the overlay image and all we need is a javascript snippet that can be called anytime and that creates a layer on top of your website with the click information. Just like the first step, we’re going to position it over the very first item in the page.
The best way to do that is via a bookmarklet, that is, an small javascript snippet saved as a bookmark. This way, you can have it in your browser and ask for the overlay image when you feel like. The javascript recalculates the offsets of the first element inside the <body> tag and writes the heatmap image on top of it.

Code: overlay.js

/**
 * @author DavidPardo
 */
//find the position of the first item on screen and store offsets
//find the first item on screen (after body)
var firstElement=document.getElementsByTagName('body')[0].childNodes[1];
//find the offset coordinates
xOffset=findPosX(firstElement);
yOffset=findPosY(firstElement);
if (IE){ // In IE there's a default margin in the page body. If margin's not defined, use defaults
	var marginLeftExplorer  = parseInt(document.getElementsByTagName('body')[0].style.marginLeft);
	var marginTopExplorer   = parseInt(document.getElementsByTagName('body')[0].style.marginTop);
	/*assume default 10px/15px margin in explorer*/
	if (isNaN(marginLeftExplorer)) {marginLeftExplorer=10;}
	if (isNaN(marginTopExplorer)) {marginTopExplorer=15;}
	xOffset=xOffset+marginLeftExplorer;
	yOffset=yOffset+marginTopExplorer;
}
// add the image element to the dom, absolutely positioned
var style="z-index:100;position:absolute;left:"+xOffset+"px;top:"+yOffset+"px";
var bodytag=document.getElementsByTagName('body')[0]
var newdiv = document.createElement('div');
var currentpage=location.href;
var escapedpage=currentpage.replace(/\W/g,'');
newdiv.setAttribute('id','heatmap');
newdiv.innerHTML = '<img src="'+escapedpage+'.final.png" style="'+style+'" >';
bodytag.appendChild(newdiv);

The result

We got a beautiful heatmap on top of our web page. We can call the overlay from wherever we want and show it to the project stakeholders. Look at the result:

The code

I made a ready to download package with all the code. It’s released under a MIT license that means that you can do whatever you want with it. Probably in the future it’ll be part of an open source release; if you feel like, start it yourself or contact me for more information.

Download code. Tar.gz file

What else?

The sky is the limit. If you want a hosted service, contact us. Our company can give you bespoke solutions to all your web intelligence needs, being it log analysis, path tracking and so on. If you’re a developer, feel free to use all the code as you wish, and please write me to tell your experiences. Stay tuned!

By the way, there has been a post in remysharp blog explaining how to record the clicks in a different server. Thanks.

Did you like it? Share us:Estos son iconos de sitios web sociales. Es posible sugerirles esta página para que la recuerden o la publiquen.

73 Comments a 'The definitive heatmap'

Subscribe to commentswith RSS o TrackBack a 'The definitive heatmap'.

Corunet. El blog » How to make heat maps dijo,

El 16 de Agosto de 2006 a las 8:44 pm

[…] NOTE: This post has been improved at The definitive heatmap There is not much documentation about creating heat maps. I haven’t been able to find an open source solution that, giving the coordinates, creates a heat map like those shown in Etre blog In the last post (part 1, part 2) we got a list of click positions in a web page, but the result is easy to improve. […]
Build a "click heat map" system using Ruby and JavaScript dijo,

El 16 de Agosto de 2006 a las 9:47 pm

[…] Heat maps allow you to see where people are clicking on your Web page(s) and while some pay-for, professional systems offer this feature, David Pardo has put together a guide, including Ruby and JavaScript source, of how to create your own from scratch! […]
Anónimo dijo,

El 16 de Agosto de 2006 a las 11:34 pm

The definitive heatmap…

What we are going to make is a complete solution that allows collecting, analyzing and showing the click information our users give us. Now, it works in web pages not center aligned and is quite a bit more robust. Read on…
Heat Maps and Usability... dijo,

El 17 de Agosto de 2006 a las 1:23 am

[…] Posted by Steve Longdo Wed, 16 Aug 2006 21:44:00 GMT I must confess I’d heard the term “heat map” in conjunction with user interaction a while ago, but never bothered to look into them in more detail. Usability pioneer Jakob Nielsen has examples in this article. Nielsen’s heat maps track human eye movement across a web page. Today I ran across this magnificent posting. It demonstrates a “heat map”, and also provides MIT licensed open source code to generate them. These “heat map” are actually “click maps”, but in some ways these are even more relevant than where a web site visitor looks. […]
Bloggitation » links for 2006-08-18 dijo,

El 18 de Agosto de 2006 a las 3:48 am

[…] The definitive heatmap Heat maps allow you to see where people are clicking on your Web page(s) and while some pay-for, professional systems offer this feature, David Pardo has put together a guide, including Ruby and JavaScript source, of how to create your own from scratch! (tags: programming ruby web apache design) […]
Daniel dijo,

El 18 de Agosto de 2006 a las 7:25 am

Hello,
Can you explain how the mod_imap works? Which script exactly calls the .map file? I see no reference to it in the downloadable files.
david dijo,

El 18 de Agosto de 2006 a las 11:49 am

Hi, Daniel

You have to replace in registerclick.js the line:

var url=’guardacoordenadas.pl?x=’+tempX+’&y=’+tempY;

For

var url=’http://mydomain.com/empty.map?x=’+tempX+’&y=’+tempY;

where empty.map is the empty .map file in your system. The sources are already updated. Sorry

About mod_imap, you can find all the documentation at http://httpd.apache.org/docs/2.0/mod/mod_imap.html

We are using it in a twisted way, but as it’s one of the core apache modules, it’s not going to dissapear soon.
Deep Jive Interests » Recommended Reads for August 18, 2006 dijo,

El 18 de Agosto de 2006 a las 2:32 pm

[…] The definitive heatmap Want to roll your own heat map so you can see where your visitors are heading? Check this article out — but you’ll need to know Javascript and Ruby to pull it off. (tags: javascript ruby cool heatmap development code) […]
WEB2.0 на Руси » Горячие точки сайта, часть 2 dijo,

El 18 de Agosto de 2006 a las 3:20 pm

[…] И вот первые попытки реализовать систему горячих точек (или карт кликов) на собственных мощностях. […]
Brain Lint » Blog Archive » Click data as heatmaps dijo,

El 18 de Agosto de 2006 a las 7:40 pm

[…] A colleague pointed out this open source project that allows users to visualize the mouse movements of users as a heatmap - the hotter the area, the more the mouse has been used there. Its a neat idea and a great effort to share the code, but I wonder about the utility of the resulting data. […]
Duncan Beevers » History » RubyInsides dijo,

El 19 de Agosto de 2006 a las 1:10 am

[…] Case-in-point, they link to this amazing heat map, which provides detailed feedback in an incredibly intuitive format. […]
jerrett dijo,

El 19 de Agosto de 2006 a las 9:31 pm

let me know if a repo is going to get setup for this , i’ve some ideas ;]
sarman dijo,

El 20 de Agosto de 2006 a las 1:01 am

Very good! All worked on my server fine. Good point to keep private data at my server=).
david dijo,

El 21 de Agosto de 2006 a las 10:19 am

Hi jerrett,

Right now the project is being reviewed at sourceforge for inclussion. Can you contact me at david@corunet.com to keep you informed via mail?
Webanalyticsbook.com » Blog Archive » Heatmaps homemade and free! dijo,

El 21 de Agosto de 2006 a las 10:21 pm

[…] The solution is completely free and is released under a MIT license. That means that you can do whatever you want with it. If you are interested read their blog and download the heatmap solutions here Posted by webmeasurement Filed in Webanalytics general […]
Web Analytique » CrazyEgg ouvre ses portes ! dijo,

El 22 de Agosto de 2006 a las 12:52 am

[…] Tout ce passe sur le site de clickmaps. […]
leo dijo,

El 23 de Agosto de 2006 a las 6:56 pm

Fantastic, I can’t wait to get working on this.

Suggestion, though. To get this working on liquid layouts, couldn’t you also send the dimensions of the viewport (using innerWidth/innerHeight ) and track those dimensions as well?

I’m not sure if you can do a prompt within a bookmarklet, but you could always create multiple bookmarklets, each for different dimensions?

Just a thought, otherwise great job.

leo
CrazyEgg Launches at Vocino.com dijo,

El 23 de Agosto de 2006 a las 7:11 pm

[…] Update: Make one yourself. […]
User Interface Metrics // Troy Gilbert dijo,

El 24 de Agosto de 2006 a las 12:27 am

[…] [Update: I found the other clickmap/heatmap site I mentioned above: The definitive heatmap..] […]
Calor, colores y preguntas - Bio XD dijo,

El 24 de Agosto de 2006 a las 7:58 am

[…] Crea tu propio heatmap con Ruby y Javascript […]
Heat Map per il proprio sito « Mac Blog dijo,

El 28 de Agosto de 2006 a las 12:23 pm

[…] Ultima opzione: se siete buoni webmaster e conoscete bene l’HTML, “Ruby on Rails” e altro createvela da voi con questo tutorial. […]
Termomapa de uso del blog | Sólo otro blog infame dijo,

El 28 de Agosto de 2006 a las 3:00 pm

[…] Si alguno queréis hacer algo más avanzado os recomiendo el artículo The definitive heatmap que me llegó a través de la recomendable lista de distribución Cadius. […]
Lain dijo,

El 30 de Agosto de 2006 a las 10:48 am

hmmm that’s interesting but what about floating layouts dependent on screen resolution??
david dijo,

El 30 de Agosto de 2006 a las 1:56 pm

Hi, Lain,

Don’t hold your breath. The only way I can think of would be a div-based page division, to create small heatmaps positioned relatively to the div corner.

That would probably be too difficult to develop in a general way, so, it’s not being taken into acount right now.
Pixel y Dixel » Heat maps: las zonas calientes de tu web dijo,

El 30 de Agosto de 2006 a las 7:54 pm

[…] Corunet ha publicado un tutorial de c�mo hacerte tu mismo tu propio heat map, usando JavaScript y Ruby. Existen tambi�n soluciones ya desarrolladas como y de pago como CrazyEgg que tan s�lo requiere insertar un c�digo en tu p�gina (hay una versi�n gratu�ta muy limitada). Tambi�n puedes probar AdGreed que es algo m�s limitada si la comparamos con CrazyEgg pero, de momento al menos, es gratuita. […]
links for 2006-08-24 at willkoca dijo,

El 1 de Septiembre de 2006 a las 8:25 am

[…] Corunet. El blog » The definitive heatmap (tags: javascript heatmap usability webdesign ruby rails) […]
Mutterings » Blog Archive » Is your site hot? dijo,

El 11 de Septiembre de 2006 a las 12:05 am

[…] Heatmap-based click analyzers are a pretty cool idea to determine how users are navigating your site. Corunet has a guide to building your own heatmaps using Javascript and Ruby and Crazy Egg has gone and built a product out of it that you can use on your own site. Nifty. […]
tazo dijo,

El 12 de Septiembre de 2006 a las 1:56 am

can you provide more documentation on how to install and use the code
david dijo,

El 12 de Septiembre de 2006 a las 10:52 am

Hi Tazo,

I’m right back from holidays. As I have reeived so many enquiries about that, I’ll try to write an installer (fully ruby based) and make it available at sourceforge, with documentation.
Original Sin dijo,

El 12 de Septiembre de 2006 a las 12:36 pm

Your map seems off. Look at the heat points to the right, it’s as if whoever clicked on the links was viewing your site with a different resolution.
It’s a nice idea though, and on the question of floating backgrounds, wouldn’t it be possible to also send back the dimentions of the browser window and then interpolate the results? It would work, unless you have a *partially* floating design, where some elements have fixed widths.
david dijo,

El 12 de Septiembre de 2006 a las 12:55 pm

Hi, Original Sin,

I did some random clicks on the page to try out the system, so, that probably explains the offset data.

About the interpolation idea, you’re right. Only trouble is that liquid layouts vary the height of the columns and all the divs (or even all the tags) would have to be treated separately.
alinush dijo,

El 14 de Septiembre de 2006 a las 5:31 pm

Simply brilliant! I can’t wait to try it out. A sourceforge release would be excellent. This will become so popular!
david dijo,

El 14 de Septiembre de 2006 a las 5:35 pm

Hi, alinush,

It’s already in sourceforge, as the project name “clickmaps”

:)
Corunet. El blog » The definitive heatmap « slow shaper dijo,

El 17 de Septiembre de 2006 a las 9:29 am

[…] Corunet. El blog » The definitive heatmap […]
lovinthemaps dijo,

El 18 de Septiembre de 2006 a las 3:41 pm

david-

great job on this. unfort. i don’t know if i can get it working as i have no access to my shared servers httpd.conf.

the mod_imap directives can be set in .htaccess, but from my understanding you cannot set CustomLog or LogFormat there.

any ideas on a workaround? or should i give up and try the old cgi version?

many thanks in any case for the fun thoughts…
Justin Blanton | Crazyegg = crazy expensive dijo,

El 30 de Septiembre de 2006 a las 9:45 pm

[…] As far as I can tell, the service offers three basic things, which are all just different presentations of the same click data: Overlay, List, and Heatmap. The “heatmap” feature is what has really set the blogosphere on fire, but is this one simple view worth $20-$100 a month? Does it really add anything to the “overlay” view, which is already available for free through Google Analytics? Does it look cool? Of course, and I’d love to use it myself, but I’ll be damned if I’m going to pay $50/mo (more than twice my actual webhosting costs) to get a pretty picture, whose ultimate significance can be derived for free from other services. Let me reiterate: save the pretty picture, this service offers nothing new. Moreover, with a little work, you can actually create a very similar picture for free, and I don’t think it’s going to be too long before someone creates a Pepper for Mint to accomplish the same. […]
Jac dijo,

El 17 de Octubre de 2006 a las 5:39 pm

Hi,

It seems like a great script, but somehow I experience problems, when I try to run it in “Ruby1.85″ I get the following error:

Invalid Parameter - -fill
Invalid Parameter - 515×77
Invalid Parameter - -negate
Invalid Parameter - -type
Invalid Parameter - -channel

Even though I have tried to read into all the great documentation, I struggle to find the solution to how to get the script to work.

Should I create images, install RMagick or some third solution?

Thanks in advance.

Jacob
links for 2006-11-16 at Blue Sky On Mars dijo,

El 16 de Noviembre de 2006 a las 11:34 am

[…] Corunet. El blog » The definitive heatmap a tool that lets you gather a heatmap for your site/app (tags: javascript usability webdesign heatmap) […]
Pedro dijo,

El 22 de Noviembre de 2006 a las 7:01 pm

Hi.
Do i need to instal RMagick for create the png and the final.png file?

Thanks.
Best Regards.
Pedro.
david dijo,

El 22 de Noviembre de 2006 a las 7:08 pm

Rmagick is needed for the last version (~30times faster) but not if you use the previous one. I’ll upload it to sourceforge as soon as possible to be ale to download the one that suits you.
Pedro dijo,

El 22 de Noviembre de 2006 a las 8:27 pm

Thank you for your quick response!

I´m receiving x, y and url values in the log file, but i can´t generate the png´s files.

My click log sample:
“?x=311&y=220,http://yd122.ydreams.com/index.htm”
“?x=234&y=199,http://yd122.ydreams.com/index.htm”
“?x=231&y=178,http://yd122.ydreams.com/index.htm”

Thanks.
Pedro Marques.
Pedro dijo,

El 22 de Noviembre de 2006 a las 8:27 pm

I´m using messenger

my adress is pedromarqueslx@hotmail.com

thanks
DLife » AJAX资料大全 dijo,

El 24 de Noviembre de 2006 a las 4:32 am

[…] Generate user-click heatmaps using JS and Ruby by David el Dia […]
130 Ajax Tutorials. - Ajax.pro dijo,

El 24 de Noviembre de 2006 a las 7:34 am

[…] Generate user-click heatmaps using JS and Ruby by David el Dia […]
Amir’s Blog » Blog Archive » A Great List of Ajax Tutorials dijo,

El 25 de Noviembre de 2006 a las 11:38 am

[…] Generate user-click heatmaps using JS and Ruby by David el Dia […]
130 Ajax Tutorials. - Ajax.pro dijo,

El 27 de Noviembre de 2006 a las 1:51 am

[…] Generate user-click heatmaps using JS and Ruby […]
Tom dijo,

El 30 de Noviembre de 2006 a las 7:12 am

I think this is a little off… I ran it and everything works great but all of my clicks are off to the right too. The math is off somewhere. Other than that very nice job!
Tom dijo,

El 30 de Noviembre de 2006 a las 7:59 am

I stand corrected… it’s right on.. My implementation was off. I Love it, thank you very much!
joe dijo,

El 30 de Noviembre de 2006 a las 1:38 pm

hi, thanks for sharing your work

For some reason Im unable to generate any clicks to the clicklog using apache2.2

I try going directly to the empty.map and it generates nothing in the log. So it has nothing to do with the javascript. It also of course doesnt work with the javascript either.

Again, i added those lines, stoped and started apache. The clicklog is created, but it just does not log anything

Im thinking maybe something has changed in apache2.2?

In the documentation they say use:

AddHandler imap-file .map

I tried both that and

AddHandler mod_imap .map

Still no go. Just wanted to see if anyone ran into a similar problem.

Thanks
Pedro dijo,

El 5 de Diciembre de 2006 a las 3:55 pm

Great work!!
I try it and run perfectly!

Thank you very much for sharing your work!!!
Pedro Marques
Austin dijo,

El 19 de Diciembre de 2006 a las 10:56 am

Wow! What excellent job putting this together.
Skylog » Blog Archive » links for 2006-12-24 dijo,

El 24 de Diciembre de 2006 a las 8:40 am

[…] The definitive heatmap (tags: internet) […]
Cubma » Blog Archive » Bygg din egen heatmap dijo,

El 2 de Enero de 2007 a las 11:26 pm

[…] Bygg din egen heatmap Att visuellt kunna se vart på ens sida besökaren klickar finns det massor av användningsområden för, speciellt inom usability. Sedan tidigare finns crazyegg och clickdensity, men det är inte säkert att man har råd att betala för en sådan tjänst då det lätt blir ganska dyrt. För den sparsame finns det nu 2 alternativ. Den första är skriven i Ruby och finns att ladda hem här, den andra är skriven i php som finns för nerladdning på labsmedia.com. Share and Enjoy:These icons link to social bookmarking sites where readers can share and discover new web pages. […]
Stef dijo,

El 10 de Enero de 2007 a las 12:48 pm

Why use imap to server the static file if you compare it to a benchmark against a .gif file there’s no optimization. Without the map_imap you can strip down apache hence it will run faster.
See tests below:

// gif
Concurrency Level: 1
Time taken for tests: 10.294763 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 3640000 bytes
HTML transferred: 430000 bytes
Requests per second: 971.37 [#/sec] (mean)
Time per request: 1.029 [ms] (mean)
Time per request: 1.029 [ms] (mean, across all concurrent requests)
Transfer rate: 345.22 [Kbytes/sec] received

// imap
Concurrency Level: 1
Time taken for tests: 10.278557 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 3540000 bytes
HTML transferred: 180000 bytes
Requests per second: 972.90 [#/sec] (mean)
Time per request: 1.028 [ms] (mean)
Time per request: 1.028 [ms] (mean, across all concurrent requests)
Transfer rate: 336.33 [Kbytes/sec] received
david dijo,

El 10 de Enero de 2007 a las 1:07 pm

It’s a perfect valid solution. You add some KBs to the transfer, but that should’t be a problem. Anyway, the best solution would be a handler that returns a “204: No Content” response and take no further action.
Ande dijo,

El 15 de Enero de 2007 a las 1:03 am

this is very useful as well as informational.. Thanks for the article and scripts examples…
How I achieved cross site scripting at remy sharp’s b:log dijo,

El 18 de Enero de 2007 a las 10:35 pm

[…] Some background: Some time ago I found, and implemented this awesome heatmap click tracking for the company I work for. However, since our test machines run everything from one machine (i.e. images and code), when it came to putting it live I never thought for a second it wouldn’t work…which was the result. […]
Carlo Robino dijo,

El 24 de Enero de 2007 a las 7:10 pm

I’m trying to realize in Java the same software in order to visualize eye-movement fixation on a web page. I have some problem to visualize the heatmap with the same graphical quality because I don’t understand what kind of graphical function has been used in Ruby…. there is someone that know how can I obtain the same graphical results by using Java?
thank you very much
Carlo
beaker dijo,

El 31 de Enero de 2007 a las 2:07 pm

very cool article. I am using it with mod_log_access, which is a drop-in replacement for mod_log_config and the clickmap logs are very useful! thanks again for an informational read.
Netzwerkkommunikation » Eyetracking ohne Tracking dijo,

El 9 de Febrero de 2007 a las 5:55 pm

[…] http://www.feng-gui.com - hier wird eine Heatmap algorithmisch erstellt. Und hier findet man ein Tool, das auf Grundlage von Webserver-Logs ebenfalls Heatmaps zeichnet. […]
metaend » Blog Archive » Feng-GUI.com / Heatmaps dijo,

El 11 de Febrero de 2007 a las 6:13 am

[…] Класиката са тези, които се слагат на готови и launched сайтове. Ето тук има напълно безплатно open source решение под MIT license. За жалост не е точно за всеки, защото човек трябва да има абсолютно всякакви права за достъп до сървъра, където се намира сайта. Има и разни платени, Crazyegg изглежда добре. […]
Scott » Blog Archive » Heatmap of system dijo,

El 14 de Febrero de 2007 a las 3:22 am

[…] http://blog.corunet.com/english/the-definitive-heatmap […]
liesen dijo,

El 19 de Febrero de 2007 a las 11:04 pm

I’ve “ported” The definitive heat map to Java. You can now create a heat map on the fly, directly with user interaction. Code and examples at: http://www.itstud.chalmers.se/~liesen/heatmap/
david dijo,

El 21 de Febrero de 2007 a las 1:28 pm

Hi Liesen,

We’ve also developed a standalone version, written in C and using LibPNG for the output for a customer. It goes about ten thousand clicks per second!
I’m glad that the ideas presented have been useful. I’ll try to write some more next month.
Max dijo,

El 20 de Marzo de 2007 a las 6:08 am

Do you have the “formula” by which you made bolilla.png? (Also, the C person that did the LibPNG port… Can you post your code?)
david dijo,

El 20 de Marzo de 2007 a las 2:33 pm

The pelotilla images were created with The Gimp, using just a gray gradient. For the LibPNG version, they were exported to PGM and filtered via a perl script. The colors were exported to BMP format and read by another script to output the vector information.

The LibPNG version isn’t meant to be published, but probably used as a web service. The only people using it in their own machines right now are the original customers (TNS Galllup) and we both share the code base.
Martin dijo,

El 21 de Marzo de 2007 a las 12:56 pm

Hi,

I was wondering whether it\’s possible to set this up without direct access to the httpd? My host, Hostgator, apparently won\’t insert the lines in the Apache httpd file as I\’m on a shared server. I guess it might affect other customers.

Any thoughts?
david dijo,

El 21 de Marzo de 2007 a las 3:50 pm

Hi martin,

You should be able to record easely the query string into a file. It can be done via PHP, Perl, Ruby… The easiest way would be (in perl)

#!/usr/bin/perl
open (LOG,”>>log.txt”);
print LOG “$ENV{QUERY_STRING}\t$ENV{HTTP_REFERER}\n”;
close (LOG)
print “Content-type: text/html\n\n”;

And that’s all (pretty basic and not production ready, but that’re the basics)
mga/blog » clic acá dijo,

El 14 de Abril de 2007 a las 6:44 am

[…] La imagen arriba muestra el patrón de clics de este blog y otro sitio web. Estos patrones son una forma de visualización de clics que se conoce como heatmap (”mapa de calor” donde las zonas más “calientes” son aquellas que tienen muchos clics). Mirando las imágenes en grande puede encontrar patrones de clic. Aunque la imagen de la izquierda presenta solamente 65 clics y la de la derecha muestra unos 480 clics se pueden observar claros patrones. Estas imágenes fueron capturadas de CrazyEgg, un servicio que permite a dueños de sitios web grabar los clics de sus visitantes por un tiempo determinado y visualizarlos. Con esta información se pueden tomar diferentes decisiones de diseño: dónde poner la publicidad, qué tipo de fotografías son más populares, cuales links son más llamativos, qué tan visible es la navegación del sitio, y realizar los cambios pertinentes para mejorar sus resultados. CrazyEgg está basado en un software libre de rastreo de clics. […]
Rick Dennis dijo,

El 2 de Mayo de 2007 a las 6:51 pm

anyone interested in setting this up on my website for some $$$$$?

please contact me….thanks.

rickdennis@hotmail.com
a heatmap? the new webcounter for your site « hakawi-tech - تكنولوجيا حكاوى dijo,

El 16 de Mayo de 2007 a las 2:26 am

[…] Link to heatmap a new way to tell who is on your site or blog. The creators of heatmap claim that it would help webmasters ‘collect, analyze and show the click information our users give us.’ Everytime a user clicks, even in an empty space on our webpage, it is reflected on the heatmap. I think the only problem with it is the slightly complicated download and installation process. […]
Open Source Heat Maps dijo,

El 4 de Junio de 2007 a las 1:38 am

[…] Corunet. El blog » The definitive heatmap After the interest shown about the clickmaps / heatmaps articles, I’ve decided to gather all the information into an easy to use system. What we are going to make is a complete solution that allows collecting, analyzing and showing the click information our users give us. Now, it works in web pages not center aligned and is quite a bit more robust. Read on… What? […]
novolocus.com » Heatmaps dijo,

El 9 de Enero de 2008 a las 7:04 pm

[…] Heatmaps are apparently the next thing in web site statistics - and what a clever idea. For a while I worked on a JS file that would report the mouse moving over, out, and clicking on DOM elements in a page, but to be honest, that was a bit excessive - and much harder to visualise. […]

Páginas

Buscar

Enlaces

Archivos

Categorías

Opciones