Corunet. El Blog » Programación http://blog.corunet.com Web development, usability and more Fri, 23 Oct 2009 15:33:27 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Twitter alerts: using twitter streaming API for fun and profit http://blog.corunet.com/twitter-alerts-using-twitter-streaming-api/ http://blog.corunet.com/twitter-alerts-using-twitter-streaming-api/#comments Fri, 23 Oct 2009 12:25:31 +0000 David Pardo http://blog.corunet.com/?p=39 twitter_alerts

Twitter is a wonderful service, but, until now, you have to subscribe to some websites to be alerted when a selected word (maybe your trademark) is tweeted. We’ll try to develop a service that filters the tweeter api, stores the interesting ones in our database, and show them in the browser in real time.

If you want to try it, watch it in action, grab the code or read on…

What?

We’ll take twitter real time results for a given word (or words) and visualize them in a browser window, like monitter.com, but on our own servers and a bit more automatic. This is going to be useful to get our own alerts and work with them.

How?

We are going to use the twitter stream API to get the comments containing a given word. Since the results are given in Json format, we’ll need to filter the data, take the interesting bits and store them. From the other side, we’re going to write a bit of javascript to read the data from the server into a web page using ajax. We could use other technologies, like comet, to push the data to the browser, but, while it would be a much cleaner implementation, I think I’m not ready to write about that yet (check this space ;) ). We need to decouple the reading of the stream from twitter and the serving to our clients because we can only have a single active  instance of the twitter streaming api at a given time, and we’re going to leave the listening proccess on for a long time using a daemon/service approach. On top of that, we want to store the tweets in a database for later perusal and data mining.

The tools

We are trying to make a useful system. To do this, we’ll need some tools:

  • HTTP server:I’m currently using Apache, but any one would be OK
  • Server side programming language: This time, PHP. While probably not as elegant or fast as Python or as cool as Ruby, gets the work done and it’s available in almost all web hosting plans. And the documentation is pretty extensive.
  • A database to store all the info
  • Client side programming: We are going to use HTML, javascript and the jQuery javascript library to simplify AJA(X) programming and for the effects.
  • A twitter acount: Yours own is OK but maybe you want to create a new one for this kind of tasks. Write down your user and password, we are going to need them soon.
  • The twitter API. Twitter people are so kind they have developed a restful API free for everyone to use. And it’s sub-zero cool.

The Twitter streaming API

Twitter has published a Streaming API that’s described as “The Twitter Streaming API allows near-realtime access to various subsets of Twitter public statuses”. In fact, this is jus what we need. You can read all the documentation at https://twitterapi.pbworks.com/Streaming-API-Documentation, but I’ll try to take the interesting parts for this project so you don’t need to yet.

We are going to use just one method (status/filter) to get results including one or several words. This method can return a stream of data in xml or json formats, has to be called using POST and can get some parameters. You can use it from the command line if you have access to some kind of unix in the following way:

curl -d 'track=google' http://stream.twitter.com/1/statuses/filter.json -uuser:password

Where user and password are your twitter credentials.

It should return something like:

response.json

Until you exit it (with CTRL+C) . This is a Json stream and can be read and parsed by several means. In fact, it’s eval-uable javascript code that we could read from the browser. But right now, we’re going to use a server-side language to read it and work with the interesting parts.

Reading Twitter stream

As I told you in the tools section, we are going to use PHP as our server side language. For the first part, the reading of the twitter stream, we don’t even need a web server, since we can run it from the command line. And if we run it from the command line, we can convert it to a kind of daemon/service and leave it on for a long time. But first, some code:

basicstreaming.php

This is (almost) the simplest code that delivers what we want. A php formatted stream of tweets that include our marked word. If you run it from your command line, it’ll show something like:

sample_tweets.php

Let’s look at the code:

We create an $opts array (in fact an array of arrays) that contain the parameters. In this particular case, we’re using two, the POST method and the search line (track=google). Then, we can treat the twitter stream as a file, using stream_context_create and fopen and just start reading lines. Each line is going to be a JSON encoded tweet, similar to what we’ve seen when we called the API from command line. Since we want to use the contents as easily as possible, we’ll need the json_decode function to parse it into PHP objects, print them and call flush just in case we’re calling the script from a browser.

Storing the results

The best way to store the results for later perusal is a database. I’m using MySQL but any other database should be OK. To be able to store the data, we need to create a database with a single table.

We are only going to store the following data:

  • Text: This is the twetter status. 140 chars max.
  • User screen name: The screen name of the poster. This is needed to create the link to twitter
  • Id: A unique id for the tweet. It’s a sequential number, so, we can order the tweets acording to this, and use it along with the user screen name to built a link back to twitter, and use it as our primary key.
  • Followers count: The number of people that are going to receive the tweet in their inboxes. We are using it to style the real-time viewer. Since my primal intention is to watch a trademark, I care about the number of people that are watching the messages.
  • The time of the tweet: basically for filtering purposes. We are going to store our server time to avoid lengthy conversions.

We could store several other fields, and a complete solution should probably take into account that you can have some different tweet types, but for the time, these four fields should suffice.

To create the table, we can run the following SQL script from the server:

createtable.sql

After that, we will have a single table database waiting for us to fill it with tweets… Let’s go:

storing_tweets_in_the_database.php

It’s as ugly as sin but it works. If you run it from the command line, it should start storing tweets in your database and keep on until you stop it. So, our database is starting to fill with tweets concerning our desired word. Now we need to be able to navigate them…

Creating the code from the server side

Now we need to publish the tweets in our browser. To do that, we need a small PHP script that returns the tweets when called. If we call it with a parameter start it’ll return all the tweets with an id bigger than that. Otherwise, it’ll return the last ten tweets stored in our database. To do that, we will use two different queries, the first one to return the last ten results, and the second one to return all results since the given id. We use subqueries (SELECT from SELECT) to get the results in our wished order.

server.php

We are going to poll this code every ten seconds and refresh the tweets list to show the most recent ones, using javascript, and the output format will be JSON.

Writing a front-end

Since, as Larry Wall said, one of the cardinal virtues of a programmer is lazyness, we are going to use jQuery  to construct the interface and the business logic. And we’re not even serving it, but linking from the Google CDN, as Dave Ward posted in his wondeful blog.

So, let’s start with the HTML:

barebones.html

Just some styles, one script link and a body containing one div, two links and a header. Simple, eh?

We need to add some javascript in the middle of the code to connect to the server:

logic.js

Let’s see… This is probably the most complex part of the article, so, I’ll try to go slow and explain every function:

getTweets(id)

This function calls the server using the getJSON jQuery method. Then, it takes each response line and calls addNew with it. If we call it with an id parameter, it’ll ask the server for all tweets with ids greater than that. Otherwise, it will grab the last ten tweets.

addNew(item)

It takes an item (a tweet) as input. It hides the first tweet, remove it’s ‘tweet’ class, appends a new tweet at the bottom and shows it. It calls the renderTweet function to get the tweet in HTML format.

renderTweet(item)

Just one line to call getImportanceColor() and a return with the HTML code. It’s a bit long but that’s because we’re adding a couple of links to the tweet to be able to visit the original one.

getImportanceColor(number)

It takes a number of followers and returns a rgb color that will be between total black, for people without followers, and total red, for Ashton Kutchner. It uses logarithms to scale between the two extremes, because there are 6 orders of magnitude between the extremes. We will use it to paint (it) black the twitters with few followers and red the twitter stars.

poll()

This is the timeout function that calls itself every 200ms and gets the new tweets.

The last block just starts the polling as soon as the document is loaded.

The Result

This is a small screen capture of my browser visiting the HTML/javascript page while running storing_tweets_in_the_database.php. It’s watching the word ‘twitter’ and, as you can see, it’s running too fast for the human eye -at least mine -, but since we are keeping all the data in our database, it’s not lost forever  :)

Limits

Right now, because of the Twitter API limits, just one instance of the watching process can be run at once. Anyhow, you can write several words, separated by commas, and Twitter will return results for all of them.This code should not be used in production, since there are almost no security checks to avoid missuse. If you want to use it in a machine open to the public, you should check -twice- every input for missbehaviour.

The code

  1. Download the code and unzip it into a folder in your local webserver
  2. Edit config.php to add your twitter login data and the words you want to watch
  3. Create the database and the table with the SQL code above
  4. Run watch.php and leave it running for as long as you wish.
  5. Visit http://localhost/thefolderwhereyouunzippedthecode/ and watch the tweets coming.

Further work

Obviously, this is just a sample. It can be made much better looking, and we could even analyze the tweets and tweet back a response to any questions concerning our keywords. The watch module should be daemonized or converted to a service to be left unatended. The HTML page could be able to filter between two dates and so on. Keep on watching. We’ll try to keep on posting this kind of contents.

Shameless plug

I’m part of Corunet, a web agency in Spain, that can deliver consistent good results in all kind of internet projects. You can visit our website http://coru.net/ or contact me at david@corunet.com if you have any special needs :)

You can follow me on twitter as @dei_biz

]]> http://blog.corunet.com/twitter-alerts-using-twitter-streaming-api/feed/ 6 Eye tracking para pobres. Parte 2 http://blog.corunet.com/eye-tracking-para-pobres-parte-2/ http://blog.corunet.com/eye-tracking-para-pobres-parte-2/#comments Mon, 24 Jul 2006 11:46:34 +0000 David Pardo http://blog.corunet.com/usabilidad/eye-tracking-para-pobres-parte-2 En la primera parte del artículo escribimos un pequeño script para analizar la posición de los clicks en una página web. Despues de unas cuantas pruebas más, ya tenemos la información necesaria para hacer un pequeño análisis.
He probado a implantar los scripts en el sitio web de corunet y he hecho algunas pruebas de clicks en la pantalla a una resolución de 1280*1024. Al cabo de 65 clicks, tengo un log como este:


x332y288 http://corunet.com/ 192.168.0.10
x399y288 http://corunet.com/ 192.168.0.10
x489y294 http://corunet.com/ 192.168.0.10
x655y346 http://corunet.com/ 192.168.0.10
x709y351 http://corunet.com/ 192.168.0.10
x350y348 http://corunet.com/ 192.168.0.10
x384y305 http://corunet.com/ 192.168.0.10
...

De aquí, me interesan las coordenadas x e y de cada click para situarlas sobre la página. Para ello, el siguiente código me va a ayudar bastante:


#!/usr/bin/perl
open (DATOS,"log.txt");
my @datos=<DATOS>;
close (DATOS);
my $i;
foreach my $dato(@datos){
if ($dato=~/\.com\/\t/){
my @posicion=split("\t",$dato);
$posicion[0]=~s/x//;
my ($x,$y)=split("y",$posicion[0]);
$i++;
print '<img src="/i/pelotilla.png"
style="z-index:'.$i.';position:absolute;
left:'.$x.'px;top:'.$y.'px;
height:16px;width:16px;">'."\n";
}
}

Este programilla (en perl) imprime un tag IMG para cada línea del log que conincide con la página raíz (termina en .com/), líneas que luego añadiremos a una copia de la página original. Además, será necesario tener una imagen en formato PNG para marcar los puntos. Yo he utilizado pelotilla.png, una imagen de 16×16px creada en photoshop.

El resultado

El programa imprime en pantalla un trocito de código que se puede pegar directamente bajo la etiqueta <body> de una copia de la página en cuestión:


...
<img src="/i/pelotilla.png"
style="z-index:51;position:absolute;
left:605px;top:773px;
height:16px;width:16px;">
<img src="/i/pelotilla.png"
style="z-index:52;position:absolute;
left:778px;top:792px;
height:16px;width:16px;">
<img src="/i/pelotilla.png"
style="z-index:53;position:absolute;
left:799px;top:810px;
height:16px;width:16px;">
...

El resultado es este, utilizando las indicaciones de webfx para conseguir transparencia en PNGs en Internet explorer:

Captura de pantalla con clicks

Cada uno de los puntos rojos se refiere a un click en la página, con lo que podemos empezar a hacernos una idea de las necesidades del usuario. Por ejemplo, en la sección de las noticias de la derecha, los usuarios han hacho click en el cuerpo de la noticia, buscando más información. Será conveniente agregar un icono de leer más, o convertir el texto en un enlace. En un análisis real, sería interesante tener más puntos, más usuarios y obtener más conclusiones…

Problemas y mejoras

  • De momento, el sistema no está preparado para cambios de resolución. En un entorno no controlado, habrá que retocar el script para que refiera las coordenadas a un punto conocido de la página.
  • El evento onclick no soporta flash. En el caso de que se use para colocar menús, será necesario editar manualmente los archivos .FLA y reemplazar los SWF para que también obtengan el punto donde se hace click

    • Partiendo de este sistema, podríamos capturar, por ejemplo, la posición del ratón cada, por ejemplo, 50 milisegundos, guardarla en un array y almacenarla en un log en el servidor, con lo que sería posible seguir los gestos del usuario tal como navega en una página web. En el futuro, más cosas.

    Agradecimientos

    He obtenido la inspiración para el artículo del post “Click Survey” que permite hacer click en una imagen determinada y analiza luego la posición de los clicks.

    ]]>     http://blog.corunet.com/eye-tracking-para-pobres-parte-2/feed/ 1     Eye tracking para pobres http://blog.corunet.com/eye-tracking-para-pobres/ http://blog.corunet.com/eye-tracking-para-pobres/#comments Mon, 17 Jul 2006 12:55:39 +0000 David Pardo http://blog.corunet.cor/general/eye-tracking-para-pobres Los dispositivos de eye tracking son caros. Dan un rendimiento excepcional para analizar adónde miran los visitantes de un sitio web, pero a veces es dificil justificar la inversión. Si consigues convencer a tus testers para que hagan click en el punto al que miran, puedes sacar conclusiones…

    Ojo de usuario

    Con un poco de ingenuidad, es posible implantar en una página web un sistema parecido, basado en javascript y que siga los puntos en donde la gente hace click con el ratón. Para ello, son necesarios los siguientes pasos:

    1. Crear una función que siga el movimiento del ratón, y que se lo pase a un logger cada vez que se hace click.
    2. Hacer una llamada a un método asíncrono (HttpRequest o similar) que mande, de forma transparente para el usuario, las coordenadas del punto en el que se ha clickado.
    3. Que el logger escriba en un archivo, base de datos o similar la información pertinente

    Capturar  la posición del ratón 

    Antes de nada, es necesario encontrar las coordenadas del ratón. Lo mejor para esto es hacer, al final del ducumento html, un enlace a un fichero javascrip parecido a este:

    /*detectar navegador y declaraciones*/
    var IE = document.all?true:false
    if (!IE) document.captureEvents(Event.MOUSEMOVE)
    var tempX = 0
    var tempY = 0/*llamada a la funcion con un handler*/
    document.onclick = getMouseXY;
    function getMouseXY(e) { //hace todo el trabajo
      if (IE) {
        tempX = event.clientX + document.body.scrollLeft
        tempY = event.clientY + document.body.scrollTop
      } else {
        tempX = e.pageX
        tempY = e.pageY
      }
      var url="/perl/guardacoordenadas.pl?x="+tempX+"&y="+tempY;
      guardar(url);
      return true
    }
    if( document.getElementsByTagName ){
     var links = document.getElementsByTagName( 'a' );
     for( var i=0; i < links.length; i++ ){
      links[i].onclick = function(){
       return getMouseXY(this);
      }
     }
    }

     Lo que hace, más o menos, es detectar que navegador está utilizando el usuario, inicializar el controlador de clicks, asignar a cada enlace una función en tiempo de ejecución y definir una función getMouseXY. Ésta devuelve true despues de llamar a una función externa que se define en el segundo paso:

    HttpRequest

    Permite hacer una llamada al servidor fuera del flujo de la navegación. En este caso, queremos que,  cuando el usuario haga click, el servidor guarde la posición del ratón. Para esto, el código es el siguiente:

    var xmlDoc = null ;
    if (typeof window.ActiveXObject != 'undefined' ) {
     xmlDoc = new ActiveXObject("Microsoft.XMLHTTP");
    }else {
     xmlDoc = new XMLHttpRequest();
    }
    function guardar(url){
     xmlDoc.open( "GET", url, true );
     xmlDoc.send( null );
    }

    Al igual que antes, diferenciamos entre el código de Internet Explorer y el resto. Luego, utilizamos el objeto xmlDoc para hacer la petición. De esta forma, llamamos al servidor sin que el cliente note nada, y podemos guardar la posición del cursor.

    El lado del servidor

    Desde el lado del servidor, la petición se ve como cualquier otra solicitud, pero no es necesario devolver nada por salida estándar. Lo único que haría falta es imprimir en un archivo la query string que le hemos pasado (con las coordenadas x e y), el referer y, si queremos, la dirección IP. Como prueba de concepto, serviría este código (CGI en perl, fácil de entender o de implementar en otro lenguaje)

    #!/usr/bin/perl
    use CGI;
    use strict;
    my $q=new CGI; #crea una nueva instancia del objeto CGI
    print $q->header(); #imprime cabeceras
    open (LOG,">>log.txt") ||die; #abre el archivo log
    print LOG $q->Vars; #e imprime todos los datos
    print LOG "\t";
    print LOG $ENV{'HTTP_REFERER'};
    print LOG "\t";
    print LOG $ENV{'REMOTE_ADDR'};
    print LOG "\n";
    close (LOG);

    Con esto, obtendremos un archivo log.txt que nos permitirá saber los puntos donde el usuario ha clickado.

    Más cosas

    Ahora sería el momento de programar un visualizador para toda esta información, superponiendo a la web los puntos en los que se ha hecho click. En breve escribiré los resultados de mi primer experimento público y como analizar todos los datos.  Ya puedes leer como analizar los logs generados en la parte 2 del artículo

    Con este mismo sistema es posible hacer muchísimas cosas. Por ejemplo, llevar un log de los enlaces internos y externos que se siguen en la página, guardar los movimientos del ratón para seguir la navegación... La imaginación es la que marca el límite.

     

     

    ]]>     http://blog.corunet.com/eye-tracking-para-pobres/feed/ 5