Corunet. El Blog » English http://blog.corunet.com Web development, usability and more Fri, 23 Oct 2009 15:33:27 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Twitter alerts: using twitter streaming API for fun and profit http://blog.corunet.com/twitter-alerts-using-twitter-streaming-api/ http://blog.corunet.com/twitter-alerts-using-twitter-streaming-api/#comments Fri, 23 Oct 2009 12:25:31 +0000 David Pardo http://blog.corunet.com/?p=39 twitter_alerts

Twitter is a wonderful service, but, until now, you have to subscribe to some websites to be alerted when a selected word (maybe your trademark) is tweeted. We’ll try to develop a service that filters the tweeter api, stores the interesting ones in our database, and show them in the browser in real time.

If you want to try it, watch it in action, grab the code or read on…

What?

We’ll take twitter real time results for a given word (or words) and visualize them in a browser window, like monitter.com, but on our own servers and a bit more automatic. This is going to be useful to get our own alerts and work with them.

How?

We are going to use the twitter stream API to get the comments containing a given word. Since the results are given in Json format, we’ll need to filter the data, take the interesting bits and store them. From the other side, we’re going to write a bit of javascript to read the data from the server into a web page using ajax. We could use other technologies, like comet, to push the data to the browser, but, while it would be a much cleaner implementation, I think I’m not ready to write about that yet (check this space ;) ). We need to decouple the reading of the stream from twitter and the serving to our clients because we can only have a single active  instance of the twitter streaming api at a given time, and we’re going to leave the listening proccess on for a long time using a daemon/service approach. On top of that, we want to store the tweets in a database for later perusal and data mining.

The tools

We are trying to make a useful system. To do this, we’ll need some tools:

  • HTTP server:I’m currently using Apache, but any one would be OK
  • Server side programming language: This time, PHP. While probably not as elegant or fast as Python or as cool as Ruby, gets the work done and it’s available in almost all web hosting plans. And the documentation is pretty extensive.
  • A database to store all the info
  • Client side programming: We are going to use HTML, javascript and the jQuery javascript library to simplify AJA(X) programming and for the effects.
  • A twitter acount: Yours own is OK but maybe you want to create a new one for this kind of tasks. Write down your user and password, we are going to need them soon.
  • The twitter API. Twitter people are so kind they have developed a restful API free for everyone to use. And it’s sub-zero cool.

The Twitter streaming API

Twitter has published a Streaming API that’s described as “The Twitter Streaming API allows near-realtime access to various subsets of Twitter public statuses”. In fact, this is jus what we need. You can read all the documentation at https://twitterapi.pbworks.com/Streaming-API-Documentation, but I’ll try to take the interesting parts for this project so you don’t need to yet.

We are going to use just one method (status/filter) to get results including one or several words. This method can return a stream of data in xml or json formats, has to be called using POST and can get some parameters. You can use it from the command line if you have access to some kind of unix in the following way:

curl -d 'track=google' http://stream.twitter.com/1/statuses/filter.json -uuser:password

Where user and password are your twitter credentials.

It should return something like:

response.json

Until you exit it (with CTRL+C) . This is a Json stream and can be read and parsed by several means. In fact, it’s eval-uable javascript code that we could read from the browser. But right now, we’re going to use a server-side language to read it and work with the interesting parts.

Reading Twitter stream

As I told you in the tools section, we are going to use PHP as our server side language. For the first part, the reading of the twitter stream, we don’t even need a web server, since we can run it from the command line. And if we run it from the command line, we can convert it to a kind of daemon/service and leave it on for a long time. But first, some code:

basicstreaming.php

This is (almost) the simplest code that delivers what we want. A php formatted stream of tweets that include our marked word. If you run it from your command line, it’ll show something like:

sample_tweets.php

Let’s look at the code:

We create an $opts array (in fact an array of arrays) that contain the parameters. In this particular case, we’re using two, the POST method and the search line (track=google). Then, we can treat the twitter stream as a file, using stream_context_create and fopen and just start reading lines. Each line is going to be a JSON encoded tweet, similar to what we’ve seen when we called the API from command line. Since we want to use the contents as easily as possible, we’ll need the json_decode function to parse it into PHP objects, print them and call flush just in case we’re calling the script from a browser.

Storing the results

The best way to store the results for later perusal is a database. I’m using MySQL but any other database should be OK. To be able to store the data, we need to create a database with a single table.

We are only going to store the following data:

  • Text: This is the twetter status. 140 chars max.
  • User screen name: The screen name of the poster. This is needed to create the link to twitter
  • Id: A unique id for the tweet. It’s a sequential number, so, we can order the tweets acording to this, and use it along with the user screen name to built a link back to twitter, and use it as our primary key.
  • Followers count: The number of people that are going to receive the tweet in their inboxes. We are using it to style the real-time viewer. Since my primal intention is to watch a trademark, I care about the number of people that are watching the messages.
  • The time of the tweet: basically for filtering purposes. We are going to store our server time to avoid lengthy conversions.

We could store several other fields, and a complete solution should probably take into account that you can have some different tweet types, but for the time, these four fields should suffice.

To create the table, we can run the following SQL script from the server:

createtable.sql

After that, we will have a single table database waiting for us to fill it with tweets… Let’s go:

storing_tweets_in_the_database.php

It’s as ugly as sin but it works. If you run it from the command line, it should start storing tweets in your database and keep on until you stop it. So, our database is starting to fill with tweets concerning our desired word. Now we need to be able to navigate them…

Creating the code from the server side

Now we need to publish the tweets in our browser. To do that, we need a small PHP script that returns the tweets when called. If we call it with a parameter start it’ll return all the tweets with an id bigger than that. Otherwise, it’ll return the last ten tweets stored in our database. To do that, we will use two different queries, the first one to return the last ten results, and the second one to return all results since the given id. We use subqueries (SELECT from SELECT) to get the results in our wished order.

server.php

We are going to poll this code every ten seconds and refresh the tweets list to show the most recent ones, using javascript, and the output format will be JSON.

Writing a front-end

Since, as Larry Wall said, one of the cardinal virtues of a programmer is lazyness, we are going to use jQuery  to construct the interface and the business logic. And we’re not even serving it, but linking from the Google CDN, as Dave Ward posted in his wondeful blog.

So, let’s start with the HTML:

barebones.html

Just some styles, one script link and a body containing one div, two links and a header. Simple, eh?

We need to add some javascript in the middle of the code to connect to the server:

logic.js

Let’s see… This is probably the most complex part of the article, so, I’ll try to go slow and explain every function:

getTweets(id)

This function calls the server using the getJSON jQuery method. Then, it takes each response line and calls addNew with it. If we call it with an id parameter, it’ll ask the server for all tweets with ids greater than that. Otherwise, it will grab the last ten tweets.

addNew(item)

It takes an item (a tweet) as input. It hides the first tweet, remove it’s ‘tweet’ class, appends a new tweet at the bottom and shows it. It calls the renderTweet function to get the tweet in HTML format.

renderTweet(item)

Just one line to call getImportanceColor() and a return with the HTML code. It’s a bit long but that’s because we’re adding a couple of links to the tweet to be able to visit the original one.

getImportanceColor(number)

It takes a number of followers and returns a rgb color that will be between total black, for people without followers, and total red, for Ashton Kutchner. It uses logarithms to scale between the two extremes, because there are 6 orders of magnitude between the extremes. We will use it to paint (it) black the twitters with few followers and red the twitter stars.

poll()

This is the timeout function that calls itself every 200ms and gets the new tweets.

The last block just starts the polling as soon as the document is loaded.

The Result

This is a small screen capture of my browser visiting the HTML/javascript page while running storing_tweets_in_the_database.php. It’s watching the word ‘twitter’ and, as you can see, it’s running too fast for the human eye -at least mine -, but since we are keeping all the data in our database, it’s not lost forever  :)

Limits

Right now, because of the Twitter API limits, just one instance of the watching process can be run at once. Anyhow, you can write several words, separated by commas, and Twitter will return results for all of them.This code should not be used in production, since there are almost no security checks to avoid missuse. If you want to use it in a machine open to the public, you should check -twice- every input for missbehaviour.

The code

  1. Download the code and unzip it into a folder in your local webserver
  2. Edit config.php to add your twitter login data and the words you want to watch
  3. Create the database and the table with the SQL code above
  4. Run watch.php and leave it running for as long as you wish.
  5. Visit http://localhost/thefolderwhereyouunzippedthecode/ and watch the tweets coming.

Further work

Obviously, this is just a sample. It can be made much better looking, and we could even analyze the tweets and tweet back a response to any questions concerning our keywords. The watch module should be daemonized or converted to a service to be left unatended. The HTML page could be able to filter between two dates and so on. Keep on watching. We’ll try to keep on posting this kind of contents.

Shameless plug

I’m part of Corunet, a web agency in Spain, that can deliver consistent good results in all kind of internet projects. You can visit our website http://coru.net/ or contact me at david@corunet.com if you have any special needs :)

You can follow me on twitter as @dei_biz

]]> http://blog.corunet.com/twitter-alerts-using-twitter-streaming-api/feed/ 6 Automatic CSS: The stylizator http://blog.corunet.com/automatic-css-the-stylizator/ http://blog.corunet.com/automatic-css-the-stylizator/#comments Wed, 20 Jun 2007 20:15:15 +0000 David Pardo http://blog.corunet.com/english/automatic-css-the-stylizator The stylizatorWriting CSS is good fun, but analyzing an html document to find how the page is structured us, at the very least, tiring. If you have ever had to write CSS for a blog or CMS template, you already know how time-consuming is to find every ID and CLASS in a large document. We’re going to write a simple script that takes an HTML file as input and gives us two things:

  • First, some CSS and Javascript to generate a visual structure with all the names like the one that can be seen at the top image (and a javascript only version!!).
  • Second, a CSS file without any rules, but with every ID and class in the document.

Let’s go:

The tools

We need to use any scripting language to read the HTML and act accordingly. This time we’re going to use perl, since it’s a simple and powerful language for text managing. It’s available in most operating systems and for windows users, it’s downloadable from http://activestate.com/ .

It should be possible to write in ruby, python, java or any other programming language. Take it as a guideline or, if you just want to use it, grab the code
We’re going to use some CSS and Javascript too. Let’s get wet…

Reading and parsing HTML

First of all, we need an HTML document to be parsed. As I’ll recall later, I used HTML code from csszengarden to try the code.
First thing to do is read the html from a file. The script we’re going starts the following way:

stylizator.1.pl

This first part opens a file, reads it and assign the contents to a variable. Now, we need to find all the id’s present in the file. To do that we could use a full-fledged HTML parser, but since many documents in the real world are not correct and our necessities are quite easy, there’s a simpler solution. So, regular expressions to the rescue.
Regular expressions find patterns in a file. We can tell perl to find things like /\w+/ and it will match alphanumeric characters. Probably to find named elements it should be enough to write:


/id=\"?\w+\"?/i

Where “id=” means exactly that, \”? means a possible quotation mark, (\w+) is a group of letters or numbers, another possible quote and the last “i” tells us to ignore the case of the text. In this particular case, we’re going to use a slightly more complicated expression, to take into acount possible missing quotes and so on:


/id\s*=\s*"?([^"\s>]+)/

Creating the CSS template

We’re going to write an empty CSS file. I like to use a previously written template that includes links to my CSS-reset file and some more files I always use. Then, for every named element, we need a line:


#namedelement{

}

stylizator.2.pl

and every time we find a class, since they are not unique, we need to filter it, creating a hash (associative array) with its name. And then, the same thing we’ve done with the IDs. Some more code:

stylizator.3.pl

With all that code, we already have a CSS template. You could start right now editing it to suit your needs. But we don’t know yet what the structure is. Why don’t we create a new HTML file that shows us how the document is laid? OK. Let’s write some more code

Javascript: The magic

Here is where the fun begins. Probably, I’ll need to wite another post about this piece of code, since there are a couple of sleights of hand, but, briefly, the following piece of code removes all the stylesheets and the inline CSS rules, then takes all the elements in the dom and finds the ones that have an ID, and adds a P element with the ID name to each one.

stylizator.js

As you can see, there are only two functions. The first one, removeSheets(), disables every stylesheet included from the HTML, and then calls the other one, findNamedElements(), This function gets all the elements in the DOM (getElementsByTagName(‘*’)) and iterates over them, doing a couple of things:

  • First, remove inline style elements (allElements[i].style.cssText = ”)
  • For the elements with an ID, some styles are added (a silver box around them)
  • A child P tag with the ID text is created too, giving it a class ’sign’

Then, we create a new stylesheet only for that class. We have to code a different way for Explorer and Firefox/Opera/Others because of compatibility issues.
Last, the function is called with removesheets(). You can even save the code as a bookmarklet and apply it to any website. It’s truly portable.

Now we need some perl code to insert the javascript into the HTML file, just before the </body>: tag.
Update: My pal Félix has just tol me that I need to add an “s” in the javascript substitution, to cope with multi-line javascript. Thanks, Félix

stylizator.4.pl

When you run the full script, another HTML file will be generated, that shows how the document is laid.

Trying it out

The best example I can think of is CSSzengarden, a site where a default layout is given to designers to skin it the better they can. I downloaded the HTML code (CC license, by-nc-sa) and called he script over it:


$> perl stylizator.pl csszengarden.html

And I got a csszengarden CSS file, with all the named tags, and a csszengarden.structure.html file with the visual structure of the site.

csszengarden.html

csszengarden.structure.html

csszengarden.css

Something like this. Click to expand(PNG 239KB):

The code

Download this file, unzip it in the same folder and call it via >perl stylizator.pl yourhtmlfile.html
It’ll create a file named yourhtmlfile.css and another one called yourhtmlfile.structure.html

Further work

There’s so much to be done. It could be possible to create a firefox extension that gives us the same visual result at the press of a button, or a system that keeps only structural CSS, discarding type and colors. The regular expressions could be better, too, to allow not-so-well-defined documents. If you have any further questions, drop me a line to david@corunet.com.

Have fun.

]]> http://blog.corunet.com/automatic-css-the-stylizator/feed/ 5 Three column layout with full page height http://blog.corunet.com/three-column-layout-with-full-page-height/ http://blog.corunet.com/three-column-layout-with-full-page-height/#comments Thu, 14 Jun 2007 19:36:56 +0000 David Pardo http://blog.corunet.com/english/three-column-layout-with-full-page-height The final result

Sometimes I wish I could do things that were easily done with table-based layouts but quite hard using just CSS.
Following a couple of posts in a CSS related Spanish mailing list (Ovillo), a guy called Zafonic showed me how to use negative margins and positive paddings to make equal sized columns. With a couple javascript functions, I patched it to fill the full browser window. Let’s see how…
Update: Error in code solved. Thanks to the testers

The problem

We’re going to write a three-column plus footer layout that covers all the browser space. The requisites are the following:

  • The three columns will have different background colors
  • We’re not going to use background images (so, no faux-columns)
  • We want the page to cover the full height of the browser. So, the footer will be at the bottom of the screen.
  • The layout has to degrade gracefully for javascript disabled browsers
  • All the code has to be valid and accesible

The beginning

The first part is simple. We need some html that defines three columns and a footer. Lets wrap the three columns within a div (called, by the way, container) and call the three columns col1, col2 and col3. The html code will be something like:

body.html

We need to give the columns a width and float them to the left. The css would be:

first.css

So, we already have a three column layout, but each column has a diferent height and the footer is far from the bottom:

first version

Negative margins to the rescue

As I already told you, I had’t heard about creating layouts using negative margins (I should have read this article from A List Apart some time ago). In a nutshell, it says that you can use a large negative margin and the opposite positive padding to “make room for the columns”. I recommend to read it if you haven’t, it’s very useful.
So, let’s go on the code. We’re going to add a -5000 pixel margin-bottom and a 5000 pixel padding-bottom to each column:

second.css

This way we get same height columns. This looks much better:

second version

but we don’t have 100% height yet. Internet explorer supports 100% height but all the other browsers don’t. So, we’ll need to use javascript to solve it:

Filling the screen – Here comes Javascript

The first thing we need is to find the height of the browser window. I use a small javascript function that takes into account the diferent behaviours of the different browsers:

height.js

This function returns the height of the browser window. It does one thing and does it well… Using this new knokledge, we can set the height of the columns, making them the browser’s window height minus the footer, to fill the screen completely. The other function we’ll need is:

fill.js

This function works by calling the last one (windowHeight) and substracting the footer size. If we’re in Explorer, we use style.height to modify the height of the columns and in all other browsers, minHeight, since Explorer doesn’t support that attribute. We’re going to call that function via onload.
And we get this:

third version

If the window is not tall enough to fit the column height, the page behaves normally:

small screen

The full code

Putting all together, this is what we get:

3.html

If you use it in a production environment, it’d be wise to split the javascript and the CSS into their own files, and review all the code, since it’s done in a quick and dirty way.

You can download the code using this link. People without javascript don’t get the full height but the site is equally usable and visually pleasant. I you have any further questions or just want to talk, drop me a line to david@corunet.com

]]> http://blog.corunet.com/three-column-layout-with-full-page-height/feed/ 3 Clickmaps in sourceforge http://blog.corunet.com/clickmaps-in-sourceforge/ http://blog.corunet.com/clickmaps-in-sourceforge/#comments Mon, 21 Aug 2006 16:55:49 +0000 David Pardo http://blog.corunet.com/english/clickmaps-in-sourceforge Today, all the code for creating clickmaps has been uploaded to sourceforge and made public under a GPL license. You can find it at http://sourceforge.net/projects/clickmaps

Thanks to jerret, the code now uses RMagick calls and it’s usable with logs sporting more than ten thousand clicks per page.

]]> http://blog.corunet.com/clickmaps-in-sourceforge/feed/ 2 The definitive heatmap http://blog.corunet.com/the-definitive-heatmap/ http://blog.corunet.com/the-definitive-heatmap/#comments Wed, 16 Aug 2006 18:39:21 +0000 David Pardo http://blog.corunet.com/english/the-definitive-heatmap The final HeatmapAfter the interest shown about the clickmaps / heatmaps articles, I’ve decided to gather all the information into an easy to use system. What we are going to make is a complete solution that allows collecting, analyzing and showing the click information our users give us. Now, it works in web pages not center aligned and is quite a bit more robust. Read on…

What?

If you are a webmaster, you had probably thought about what do users do in your website. Beyond usual statistics, clickmaps allow you to find where your users are clicking. This is quite useful to find areas in needing of change, layouts that don’t work as intended or anchors that aren’t being understood as you would like.

You’re going to be able to find every single click your users make in your website, being over a link or even in blank areas. We are going to do it the following way:

The proccess

We need to divide the full proccess into some manageable steps that use some open source tools. Since I work both in windows and linux systems, I’ll be OS agnostic and use only tools available in most systems, including Mac OSX.
The main steps and the tools they use are the following:

  1. The collecting (javascript and apache)
  2. The processing (ruby and imageMagick)
  3. The showing (javascript)

The collecting

We are going to use a small snippet of unobtrusive javascript to allow the client to tell our server the click positions. Just place this small javascript file at the very end of your template, right before the closing <body> tag:

registerclicks.js

The code adds a onMouseDown handler to the document, executes a function for every click and returns true, since we want the user to follow the normal navigation. Then, when the user clicks any part of the page, a tiny request is going to get sent on the background to our server. The script has to calculate the offsett of the first element inside the <body> tag, because most pages arent aligned to the top-right corner. In liquid layouts the system is not going to work at all

The request is sent via a HttpRequest object that calls a file in the server. In last version, I used a small GCI written in perl to log the request and return an empty document, but since we want to serve so many request, there’s a better method to apply. Using a perl CGI, in a modern server, we get the following results benchmarking with apache bench (100 requests, 10 concurrent ones):


Concurrency Level: 10
Time taken for tests: 6.537187 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 17100 bytes
HTML transferred: 0 bytes
Requests per second: 15.30 [#/sec] (mean)
Time per request: 653.719 [ms] (mean)
Time per request: 65.372 [ms] (mean, across ...)
Transfer rate: 2.45 [Kbytes/sec] received

Mod_imap

Apache has some modules that work the following way:
You define a handler and what you want to do with it. Some of them are well known, like mod_perl or mod_cgi, but a lesser known one, called mod_imap, does exactly what we want. It’s a module meant to return server-side image maps, but if we use an empty map file, all we get is a 204 status (no data) and a logged transaction. The difference is quite significative. Using Apache Bench with the same configuration, this is what we get:


Concurrency Level: 10
Time taken for tests: 0.106316 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 36464 bytes
HTML transferred: 20246 bytes
Requests per second: 940.59 [#/sec] (mean)
Time per request: 10.632 [ms] (mean)
Time per request: 1.063 [ms] (mean, across ...)
Transfer rate: 329.21 [Kbytes/sec] received

That’s 950 requests per second vs 15 with the CGI method!!! We are almost a hundred times faster with this approach!The only thing we have to do to use this mod_imap is to touch a little bit the apache configuration file. Do it carefully because it can hurt your entire server. In the relevant section add the following lines:


AddHandler mod_imap .map
CustomLog /tmp/clicklog clicklog #or modify according to your system

And define a custom log in the same file adding this:

LogFormat "%q,%{Referer}i" clicklog

This way, everything ending in .map is going to be treated as a server-side map, and since the map is empty, it’s not taking your user anywhere. But it logs it, in file /tmp/clicklog (YMMV).

The log analysis

Since we used a logFormat apache directive to write our log, the format should be easy to parse. The query string is written in the log as it comes, and the full lines should be in the following format:


?x=483&y=32&dx10&dy15,http://demo.html
?x=461&y=177&dx10&dy15,http://demo.html
?x=408&y=40&dx10&dy15,http://demo.html
(...)

I decided to write a Ruby script to parse the file and generate the final images, because I hadn’t used ruby before and thought it would be a good way to approach the problem. Last time I had written an structured perl script, but I think that object-oriented is the way to go in this particular situation, since the objects should be well-defined and dividing the program among several coders should be easier too.

Update:Thanks to Jerret, this part has been enhanced using RMagick. Part of the code below can be updated and works some 50 times faster. On top of that, a new sourceforge project has been started at http://sourceforge.net/projects/clickmaps/ under a GPL license. Of course, if you don’t want to install/use RMagick you can still download the original version at the end of this post.

I´ll try to explain the model. It uses five classes:

Conf:
Sets some configuration variables and returns them as a hash. This way, every configuration variable is set in this class and it’s easy to get them later on

conf.rb

Readparsefile
Reads and parses the file defined as logfile in the conf object. For each log line, it stores it into a click object and append it to an array. There are two methods that return all the URLs in the log file (geturls) and all the information for a single URL as a Log object

readparsefile.rb

Click
Stores the data in each log line, including X, Y and URL. Provides a method (xy) that returns an string like “x100y200″ to compare the exact coordinates, useful to extract the maximum number of times a single click is repeated

click.rb

Log
Stores all the values pertinent to a single URL and gives accesors to them. There’s also a “next” method that returns next click within the same URL

log.rb

Image
Receives a log object and the conf object. There are three methods to normalize the spot we’re going to use as a click indicator (normalizespot), compose every click as a dot (iterate) and colorize the final image (colorize)

image.rb

Then, the main program is only eight lines long. It leverages the objects’ methods to be as compact as possible. In fact, the only thing it does is to iterate over each url to create a different image.

conf = Conf.new
file = Readparsefile.new(conf.data['logfile'])
file.geturls.each do |url|
    image = Image.new(file.coordsurl(url),conf)
    image.normalizespot
    image.iterate
    image.colorize
end

Is it better?

You can find another program (this time written in perl) in an older post that does a similar job of making heatmaps. But there has been some modifications that makes this an usable system instead of a proof of concept:

Flexible configuration
Over the harcoded last version, in this one is quite simple to modify the images used in the heatmap generation, or the log name. You only have to modify the Conf definition. It would be so easy to use an external conf file, but doing it this way is quicker for me
Multiple URL support
While last version only let you extract one image, this one makes a heatmap for every URL in your log.
Much faster execution time
Instead of composing the full image everytime, now we create a single ImageMagick sentence to do al the composition for us. That gives us a couple of orders speed advantage. Last version lasted about fifteen minutes for a couple hundred clicks, and now it’s about five seconds. Please note that, for many clicks, the program uses quite a bit of memory. Probably for a production environment it would be neccesary to divide the compose sentence into manageable chuncks, and iterate at the end with them to create the final heatmap.
Manual capture is not needed anymore
Since the last step is to decrement the opacity of the map, we can use a little bit of javascript to overlay the PNG image over the original page. So, the stakeholders can review it without someone manually capturing the screen. This way we don’t need to set an XServer in the production environment
Easier to maintain and extend
The object oriented paradigm doesn’t give us faster code, but much more manageable one. You can extend it as you want

What you get

Now, you’ll have several images. Most of them are OK to delete, but there’s one ending in final.png that’s your heatmap. We’re going to overlay it on top of your web page. That image should be a semi-transparent PNG like this one:

The overlay

This is the final part of the proccess. We already have the overlay image and all we need is a javascript snippet that can be called anytime and that creates a layer on top of your website with the click information. Just like the first step, we’re going to position it over the very first item in the page.
The best way to do that is via a bookmarklet, that is, an small javascript snippet saved as a bookmark. This way, you can have it in your browser and ask for the overlay image when you feel like. The javascript recalculates the offsets of the first element inside the <body> tag and writes the heatmap image on top of it.

overlay.js

The result

We got a beautiful heatmap on top of our web page. We can call the overlay from wherever we want and show it to the project stakeholders. Look at the result:

The code

I made a ready to download package with all the code. It’s released under a MIT license that means that you can do whatever you want with it. Probably in the future it’ll be part of an open source release; if you feel like, start it yourself or contact me for more information.

Download code. Tar.gz file

What else?

The sky is the limit. If you want a hosted service, contact us. Our company can give you bespoke solutions to all your web intelligence needs, being it log analysis, path tracking and so on. If you’re a developer, feel free to use all the code as you wish, and please write me to tell your experiences. Stay tuned!

By the way, there has been a post in remysharp blog explaining how to record the clicks in a different server. Thanks.

]]> http://blog.corunet.com/the-definitive-heatmap/feed/ 81 How to make heat maps http://blog.corunet.com/how-to-make-heat-maps/ http://blog.corunet.com/how-to-make-heat-maps/#comments Sun, 06 Aug 2006 16:30:37 +0000 David Pardo http://blog.corunet.com/english/how-to-make-heat-maps NOTE: This post has been improved at The definitive heatmap
There is not much documentation about creating heat maps. I haven’t been able to find an open source solution that, giving the coordinates, creates a heat map like those shown in Etre blog
In the last post (part 1, part 2) we got a list of click positions in a web page, but the result is easy to improve.


We have the tools. Using a bit of scripting and ImageMagick, it shouldn’t be difficult to make a useful heat map.

The steps

  1. Normalizing the data
  2. Plotting the dots
  3. Capturing the screen
  4. Making the overlay

Normalizing the data

Lets assume that the data comes in the following format:

x332y288 http://corunet.com/ 192.168.0.10
x399y288 http://corunet.com/ 192.168.0.10
x489y294 http://corunet.com/ 192.168.0.10
x655y346 http://corunet.com/ 192.168.0.10
x709y351 http://corunet.com/ 192.168.0.10
x350y348 http://corunet.com/ 192.168.0.10
x384y305 http://corunet.com/ 192.168.0.10
…

We need to pick the coordinates and find the boundary values (maximum) and the maximum number of clicks in the same point. This way, we will use that number as the maximum intensity in our heat map. The point most reclicked will have an intensity of 255 and the dots with zero clicks will have zero value. This way, we will probably exceed the maximum value for dots close to each other, but we’ll try to solve it later..

A better way to do it would be to define a matrix as big as the maximum values, add the coordinates and normalyze it, making the biggest value 255 and the smallest one zero, but that will have to wait, at least, until the second version.

The first part of the code (again, in perl) will be like this, marking split lines with #//:


#!/usr/bin/perl
use strict;
my $imgWidth=64;
open (LOG,"log.txt") || die;
my @log=<LOG>;
close (LOG);
my (@x,@y);
my ($xMax,$yMax,$repetitionsMax,$i,$j);
my %repetitions;
foreach my $line(@log){
(my $coords,undef,undef)=split("\t",$line);
$repetitions{$coords}++;
$coords=~s/x//;
($x[$i],$y[$i])=split("y",$coords);
$i++;
}
while ((my $key, my $value)=each(%repetitions)){
$repetitionsMax=$value if $repetitionsMax<$value;
}
for ($j=0;$j<$i;$j++){
$xMax=$x[$j] if $xMax<$x[$j];
$yMax=$y[$j] if $yMax<$y[$j];
}
print $repetitionsMax,"\n$xMax;$yMax\n";

Now, we have three interesting values: xMax, yMax and repetitionsMax. The first two are going to be the dimensions of the canvas, and the third one, our maximum value. For me, those are 848 pixels, 2641 pixels and 3 repetitions. So, lets create the canvas:


my $xCanvas=$xMax+int($imgWidth/2);
my $yCanvas=$yMax+int($imgWidth/2);
my $createcanvas="convert -size ". $xCanvas.#//
"x".$yCanvas." pattern:gray100 empty.png";
system ($createcanvas);

Plotting the dots

We are going to compound our recently created canvas with an existing image for every line in the log. The image, bolilla.png, is a PNG file with transparency that allows us to paint a region instead of a single dot for each click. As the maximum opacity value is 1 (the center of the image is completely opaque) we will use a simple formula to determine the percent of overlay we are going to use: Percent=100/repetitionsMax:


my $createNormalizedSpot="convert bolilla.png -fill #//
white -colorize ".int(100/$repetitionsMax)."% bol.png";
system ($createNormalizedSpot);

So, for each log line, I’ll compound the image again over the canvas:


for ($j=0;$j<$i;$j++){
my ($x,$y);
$x=$x[$j]-32;
$y=$y[$j]-32;
system ("composite -compose multiply -geometry #//
+$x+$y bol.png empty.png empty.png\n");
print "used $j of $i clicks\n";
}

And I got something like this:

Making the final image

We need to remap empty.png to a false color colormap to get the typical heat map appearance. The first thing that come to my mind was to use photoshop, but it’s much more interesting to do it in Imagemagick too. Version 6 is needed to apply colors from one image to another, but once I discovered that, worked like a charm. First of all, we need to invert the image we got:


convert empty.png -negate full.png

And then, replace the colors with


convert full.png colors.png -fx "v.p{0,u*v.h}" final.png

Final map image

Where colors.png is a gradient image i made first. You can download and use it as you want.

Capturing the screen

The cheapest and easiest way to capture the original web screen is by using Firefox’ Screen Grab. It allows us to capture the complete web page, without having to capture it screen by screen and stitching by hand. What we get is something like this:

screen capture

I saved the result as blog.png

Compounding the final image

Now, all we have to do is to compound the final image with the screen capture, again from the command line:


composite -blend 40% final.png blog.png heatmap.png

The result

What we got is a heat map of the data in the log. It’s a bit more visual than the last version, and easier to understand when we have lots of dots.

Final heatmap

The code

You can download the perl script that does all the work. You will need a working instance of perl (5.6 or better) and ImageMagick V6. Both of them should come as standard in moder linux distributions, or you can install each in a windows system from Activestate (perl) and ImageMagick.org (ImageMagick).

Download code. tgz file, 258KB.

Drop me a line if you use it!

david@corunet.com

]]> http://blog.corunet.com/how-to-make-heat-maps/feed/ 5 Zero budget eye tracking. Clickmaps part 2 http://blog.corunet.com/zero-budget-eye-tracking-clickmaps-part-2/ http://blog.corunet.com/zero-budget-eye-tracking-clickmaps-part-2/#comments Sun, 30 Jul 2006 16:01:08 +0000 David Pardo http://blog.corunet.com/english/zero-budget-eye-tracking-clickmaps-part-2 In the first part of Zero budget eye tracking, we wrote a small script to keep track of the clicks in a web page. A couple of days after, we already have some information to analyze. Keep on reading to know how…
NOTE: This post has been improved at The definitive heatmap
I added the scripts to our website (www.corunet.com) and used a resolution of 1280×1024 px. After 65 clicks, I have a log like this one:


x332y288 http://corunet.com/ 192.168.0.10
x399y288 http://corunet.com/ 192.168.0.10
x489y294 http://corunet.com/ 192.168.0.10
x655y346 http://corunet.com/ 192.168.0.10
x709y351 http://corunet.com/ 192.168.0.10
x350y348 http://corunet.com/ 192.168.0.10
x384y305 http://corunet.com/ 192.168.0.10
...

We would need to filter the x and y coordinates to draw some dots over the page. I will use the following code (written in perl):


#!/usr/bin/perl
open (DATOS,"log.txt");
my @datos=<DATOS>;
close (DATOS);
my $i;
foreach my $dato(@datos){
if ($dato=~/\.com\/\t/){
my @posicion=split("\t",$dato);
$posicion[0]=~s/x//;
my ($x,$y)=split("y",$posicion[0]);
$i++;
print '<img src="/i/pelotilla.png"
style="z-index:'.$i.';position:absolute;
left:'.$x.'px;top:'.$y.'px;
height:16px;width:16px;">'."\n";
}
}

This script writes out an IMG tag for every log line that asked for the main page (ends in .com). This way, we would only need to paste this html code into a copy of the original page. We also will need a small image to mark the spots. I used a 16×16px

The click map

The former script prints out some html code like this one:


...
<img src="/i/pelotilla.png"
style="z-index:51;position:absolute;
left:605px;top:773px;
height:16px;width:16px;">
<img src="/i/pelotilla.png"
style="z-index:52;position:absolute;
left:778px;top:792px;
height:16px;width:16px;">
<img src="/i/pelotilla.png"
style="z-index:53;position:absolute;
left:799px;top:810px;
height:16px;width:16px;">
...

Pasting the code into the page copy, this is what we get:

Click map

I used the system posted in webfx to use transparent PNGs in internet explorer. We wouldn’t need it if we had done our research using Opera or Firefox browsers.
Each dot in the above image refers to a single click in our web site. Studying it we would be able to identify areas where the users are having trouble to find the links, and where they think should be a link but isn’t. In a real world analysis it would be better to have some more dots to work on.

Shortcomings

  • The system’s not able to support resolution changes. We would need some more javascript to refer every click to a known point of the screen. And it would be pretty difficult, if not impossible, to use it in liquid layouts.
  • In flash movies, we are going to have some trouble because the onclick event is not supported. As you can see in the screen capture above, there is not a single click detected in the top area, where the flash menu is.

    • Using the same technique, we could be able to store every mouse position in some milliseconds interval to replay our users movements, and there are many other possibilities.

    Acknowledgements

    I got the inspiration for this post from the “Click Survey” that lets you click randomly in any position of a picture.

    ]]>     http://blog.corunet.com/zero-budget-eye-tracking-clickmaps-part-2/feed/ 0     Zero budget eye tracking. Clickmaps. http://blog.corunet.com/zero-budget-eye-tracking-clickmaps/ http://blog.corunet.com/zero-budget-eye-tracking-clickmaps/#comments Fri, 28 Jul 2006 12:50:18 +0000 David Pardo http://blog.corunet.com/english/zero-budget-eye-tracking-clickmaps Eye tracking is quite expensive. The hardware and software have astounding capabilities, but sometimes it’s hard to justify the expenditure. But there is a technique that can imitate it until certain point. You can find where your users are clicking using just a bit of javascript and server side programming. Find out how…
    NOTE: This post has been improved at The definitive heatmap

    Ojo de usuario

    We need to follow three simple steps:

    1. Create a function that follows the mouse movement and calls an external script in the server for every click
    2. Call the server using an asynchronous method like HttpRequest
    3. Use a logging program to write a log in the server

    Capturing the mouse position

    The first thing to do is to find out the mouse position. The best way to approach this is to link a javascript file from the end of your html document. The file should be something like this:

    /*detect browser*/
    var IE = document.all?true:false
    if (!IE) document.captureEvents(Event.MOUSEMOVE)
    var tempX = 0
    var tempY = 0
    /*call the function*/
    document.onclick = getMouseXY;
    function getMouseXY(e) {
    if (IE) {
    tempX = event.clientX + document.body.scrollLeft
    tempY = event.clientY + document.body.scrollTop
    } else {
    tempX = e.pageX
    tempY = e.pageY
    }
    var url="/perl/guardacoordenadas.pl?x="+tempX+"&y="+tempY;
    guardar(url);
    return true
    }
    if( document.getElementsByTagName ){
    var links = document.getElementsByTagName( 'a' );
    for( var i=0; i < links.length; i++ ){
    links[i].onclick = function(){
    return getMouseXY(this);
    }
    }
    }

    This script filters the browser, inits the click controller and calls an external function guardar that will use the HttpRequest object to call the server.

    HttpRequest

    The HttpRequest object allows us to call the server outside the normal navigation flow. What we want now is to store the mouse position every time the user clicks in the page. We are going to use the following javascript code:

    var xmlDoc = null ;
    if (typeof window.ActiveXObject != 'undefined' ) {
    xmlDoc = new ActiveXObject("Microsoft.XMLHTTP");
    }else {
    xmlDoc = new XMLHttpRequest();
    }
    function guardar(url){
    xmlDoc.open( "GET", url, true );
    xmlDoc.send( null );
    }

    Just like before, we have to use a different approach for internet explorer and all other browsers, since explorer uses its own implementation, to create an asynchronous call object (xmlDoc). Then, we use this object to call the server in the background using a GET method to send the mouse position.

    The server side

    From the server side, there's no need to send anything to the user. We only need to store the variables sent by the query string in a file (or database), mainly the coordinates, the referer and the IP address. The following perl/CGI code should work as a proof of concept

    #!/usr/bin/perl
    use CGI;
    use strict;
    my $q=new CGI; #create a new GCI instance
    print $q->header(); #print headers
    open (LOG,">>log.txt") ||die; #open log file to append
    print LOG $q->Vars; #and prints all the data
    print LOG "\t";
    print LOG $ENV{'HTTP_REFERER'};
    print LOG "\t";
    print LOG $ENV{'REMOTE_ADDR'};
    print LOG "\n";
    close (LOG);

    This script willl write a line for every click in our page.

    To do

    Now it's time to analyze the data we got. We would need to filter the log and make a clickmap. The English version of the second part is still unavailable, but you can read the Spanish version if you want. I'll try to translate it as soon as possible

    There are many other things that can be done with this approach. You can, for example, write a log with every mouse movement, to keep track of the users' navigation. If you find some creative uses for this, please comment.

    ]]>     http://blog.corunet.com/zero-budget-eye-tracking-clickmaps/feed/ 12