Monday, February 25, 2013

D3js Electoral map

After trying to draw an electoral map using Kartograph, this time I've tried with D3js.
This has some good things, such as being able to use topoJSON which reduces dramatically the size of the files or using all the visualization tools included in D3js. On the other hand, Kartogrph has some good styling aids that help a lot.

As usual, you can download all the source code
or take a look to the examples:
Simple Map -- source code
Select Order Map -- source code
Simple Tooltip Map -- source code
Pie Chart Tooltip Map -- source code

Simple map

Let's start with the basic choropleth map:
The complete code for the example can be found here.
The image will be an SVG, so web can add interactivity and style it easily. 
The scripts included will be d3js, of course, and topojson, since this is the format of the data (see the last point):

 
The JavaScript part, then is:
var width = 600,
    height = 600;

var projection = d3.geo.mercator()
    .center([2,41.5])
    .scale(50000)
    .translate([width / 2, height / 2]);

var path = d3.geo.path()
    .projection(projection);

var svg = d3.select("#map").append("svg")
    .attr("width", width)
    .attr("height", height);

d3.json("mun_out_topo.json", function(error, topo) {
  svg.selectAll()
      .data(topojson.object(topo, topo.objects.mun_out).geometries)
    .enter().append("path")
      .attr("class", function(d) {
           var maxVotes = 0;
           var party = null;
           if (d.properties["CiU"]>maxVotes){maxVotes = d.properties["CiU"]; party="CiU";}
           if (d.properties["PSC"]>maxVotes){maxVotes = d.properties["PSC"]; party="PSC";}
           if (d.properties["ERC"]>maxVotes){maxVotes = d.properties["ERC"]; party="ERC";}
           if (d.properties["ICV-EUiA"]>maxVotes){maxVotes = d.properties["ICV-EUiA"]; party="ICV-EUiA";}
           if (d.properties["CUP"]>maxVotes){maxVotes = d.properties["CUP"]; party="CUP";}
           if (d.properties["C's"]>maxVotes){maxVotes = d.properties["C's"]; party="Cs";}
           return "municipality " + party; 
       })
      .attr("d", path);
});

  1. See, at line 4, how the projection is set. I have chosen Mecator and centered it at the coordinates I know are more or less at the center of my bounding box. Then with a lot of patience, just trying, I have found that the scale 50000 is the one that fits better for the image size.
  2. At line 9, the path object is set, and then at line 12, the SVG object is assigned to the div with the id=map. I prefer to put it into a div rather than directly to the body tag, as most of the d3js examples do.
  3. At line 16, the topoJSON is loaded, and the other stuff is run only after this is done. Do not put this code outside the function or it won't work.
  4. At line 17, the map drawing starts:
    1. The topoJSON elements are assigned as the data. In this case, the name of the elements is topo.objects.mun_out, but to find it, the best is to look directly into the topoJSON file.
    2. With the enter() method, the following methods will be applied to every element. The first thing done is appending a path element to the svg.
    3. The class is set, so the map can have different colours depending on the winning party. To do it, the function looks into the element properties. Again, take a look into the topoJSON file to see how the information is stored. The string returned is municipality and the winner party. The css at the header of the file sets the colour for the background and stroke.
    4. Finally, the path of the element is set to the svg, actually drawing the shape.

Selecting the order

The map above shows only the party that won in every municipality. What about changing that, choosing the position the user wants to show? With d3js is quite easy to do it.

The complete code for the example can be found here.
The part changed from the first example is after loading the topoJSON file:

d3.json("mun_out_topo.json", function(error, topo) {
  svg.selectAll("municipality")
      .data(topojson.object(topo, topo.objects.mun_out).geometries)
    .enter().append("path")
      .attr("class", function(d) {return "municipality " + selectParty(d,1);})
      .attr("d", path);
       
      function selectParty(d,position){
   
           var positions = new Array();
           positions[0] = parseInt(d.properties["CiU"]);
           positions[1] = parseInt(d.properties["PSC"]);
           positions[2] = parseInt(d.properties["ERC"]);
           positions[3] = parseInt(d.properties["PP"]);
           positions[4] = parseInt(d.properties["ICV-EUiA"]);
           positions[5] = parseInt(d.properties["CUP"]);
           positions[6] = parseInt(d.properties["C's"]);
 
           positions.sort(function(a,b) { return b-a; });
            
           var party = null;
           if (positions[position-1] == parseInt(d.properties["CiU"])){
               party = "CiU";
           } else if (positions[position-1] == parseInt(d.properties["PSC"])){
               party = "PSC";              
           } else if (positions[position-1] == parseInt(d.properties["ERC"])){
               party = "ERC";              
           } else if (positions[position-1] == parseInt(d.properties["PP"])){
               party = "PP";              
           } else if (positions[position-1] == parseInt(d.properties["ICV-EUiA"])){
               party = "ICV-EUiA";              
           } else if (positions[position-1] == parseInt(d.properties["CUP"])){
               party = "CUP";              
           } else if (positions[position-1] == parseInt(d.properties["C's"])){
               party = "Cs";              
           }
     
           return party;
      }
       
      d3.select("#position").on("change", function() {
            
           var position = parseInt(this.value);
           svg.transition()
           .selectAll(".municipality")
           .attr("class", function(d) {return "municipality " + selectParty(d,position);});
      });
 
});

  1.  At line 8, note that a new function is defined. The function returns the class name depending on the order position, passed as a parameter. At line 5, the function is called for the first time, asking for the first position.
  2. At line 41, an event method is added. When the selector changes its value, a transition is passed to the svg, calling the selectParty method to re-calculate the classes.
As you can see, modifying the properties of all the svg objects is quite simple.

Adding tooltips

Showing the results for a selected municipality when the mouse is over is also quite simple and improves a lot the map.
The complete code of the example can be found here

The tooltip is created using the files from this example. (although styled to make it contrast a little, and commenting the line 20)

d3.json("mun_out_topo.json", function(error, topo) {
  svg.selectAll("municipality")
      .data(topojson.object(topo, topo.objects.mun_out).geometries)
    .enter().append("path")
      .attr("class", function(d) {return "municipality " + selectParty(d,1);})
      .attr("d", path)
      .call(d3.helper.tooltip(function(d, i){return tooltipText(d);}));
      
      function selectParty(d,position){
  
           var positions = new Array();
           positions[0] = parseInt(d.properties["CiU"]);
           positions[1] = parseInt(d.properties["PSC"]);
           positions[2] = parseInt(d.properties["ERC"]);
           positions[3] = parseInt(d.properties["PP"]);
           positions[4] = parseInt(d.properties["ICV-EUiA"]);
           positions[5] = parseInt(d.properties["CUP"]);
           positions[6] = parseInt(d.properties["C's"]);

           positions.sort(function(a,b) { return b-a; });
           
           var party = null;
           if (positions[position-1] == parseInt(d.properties["CiU"])){
               party = "CiU";
           } else if (positions[position-1] == parseInt(d.properties["PSC"])){
               party = "PSC";               
           } else if (positions[position-1] == parseInt(d.properties["ERC"])){
               party = "ERC";               
           } else if (positions[position-1] == parseInt(d.properties["PP"])){
               party = "PP";               
           } else if (positions[position-1] == parseInt(d.properties["ICV-EUiA"])){
               party = "ICV-EUiA";               
           } else if (positions[position-1] == parseInt(d.properties["CUP"])){
               party = "CUP";               
           } else if (positions[position-1] == parseInt(d.properties["C's"])){
               party = "Cs";               
           }
    
           return party;
      }
      
      function tooltipText(d){
           return "" + d.properties["Name"] + ""
                  + "
 CiU: " + d.properties["CiU"] 
                  + "
 PSC: " + d.properties["PSC"]
                  + "
 ERC: " + d.properties["ERC"]
                  + "
 PP: " + d.properties["PP"]
                  + "
 ICV-EUiA: " + d.properties["ICV-EUiA"]
                  + "
 CUP: " + d.properties["CUP"]
                  + "
 C's: " + d.properties["C's"];
      }
      d3.select("#position").on("change", function() {
           
           var position = parseInt(this.value);
           svg.transition()
           .selectAll(".municipality")
           .attr("class", function(d) {return "municipality " + selectParty(d,position);});
      });

});
Again, the code needs only small changes:
  1. At line 7, the event is added to each feature. I have separated the text generation into a function to make it easier to understand.
  2. At line  42 the function tooltipText is defined. It just returns the desired text getting all the properties from each feature.

Cool tooltips using d3js

The best thing about using d3js is that you can mix all its visual possibilities, which are infinite. In the electoral map case, a donut chart helps a lot when interpreting the numbers, at least, much more than showing only the number of votes, that cchange a lot in every municipality.
The complete code for the example can be found  here

I have taken the donut chart code from this example, and the label positions from this other example.

d3.helper = {};
d3.helper.tooltip = function (accessor){
    return function(selection){
 var tooltipDiv;
        var bodyNode = d3.select('body').node();
        selection.on("mouseover", function(d, i){
            d3.select('body').selectAll('div.tooltip').remove();
            tooltipDiv = d3.select('body').append('div').attr('class', 'tooltip');
            var absoluteMousePos = d3.mouse(bodyNode);
            tooltipDiv.style('left', (absoluteMousePos[0] + 10)+'px')
                .style('top', (absoluteMousePos[1] - 15)+'px')
                .style('position', 'absolute') 
                .style('z-index', 1001);
            var arc = d3.svg.arc()
                .outerRadius(120)
                .innerRadius(40);

            var pie = d3.layout.pie()
               .sort(null)
               .value(function(d) { return d.votes; });

            var svg = tooltipDiv.append("svg")
                .attr("width", 270)
                .attr("height", 300)
                .append("g")
                .attr("transform", "translate(" + 270 / 2 + "," + 270 / 2 + ")");
 
            var data = [
                {'party':"CiU",'votes':d.properties["CiU"]},
                {'party':"PSC",'votes':d.properties["PSC"]},
                {'party':"ERC",'votes':d.properties["ERC"]},
                {'party':"PP",'votes':d.properties["PP"]},
                {'party':"ICV",'votes':d.properties["ICV-EUiA"]},
                {'party':"CUP",'votes':d.properties["CUP"]}, 
                {'party':"C's",'votes':d.properties["C's"]}
            ];
            data.forEach(function(d) {
                d.votes = +d.votes;
            });

  

            var g = svg.selectAll(".arc")
               .data(pie(data))
               .enter().append("g")
               .attr("class", "arc");

            g.append("path")
              .attr("d", arc)
              .style("fill", function(d) { return color(d.data.party); });
            g.append("text")
              .attr("transform", function(d) { var angle =(180/Math.PI) * (d.startAngle + (d.endAngle-d.startAngle)/2); return "translate(" + arc.centroid(d) + ") rotate("+angle+", 0,0)"; })
              .attr("dy", "-2.5em")
              .style("text-anchor", "middle")
              .text(function(d) { return d.data.party; });

  

          var municipality = d.properties['Name'];
          
          svg.append("text")
              .attr("transform", "translate(0,140)")
              .attr("dy", ".35em")
              .style("text-anchor", "middle")
              .text(municipality);
            
                      
        })
        .on('mousemove', function(d, i) {
            var absoluteMousePos = d3.mouse(bodyNode);
            tooltipDiv.style('left', (absoluteMousePos[0] + 10)+'px')
                .style('top', (absoluteMousePos[1] - 15)+'px');
            var tooltipText = accessor(d, i) || '';
            //tooltipDiv.html(tooltipText);
            
        })
        .on("mouseout", function(d, i){
            tooltipDiv.remove();
        });
    };    
};
This piece of code is put before loading the topoJSON. Is more or less this example, adapted to show the parties results (line 28) and puting the labels using an angle (line 52). Notice that first, the rotation is done, and only then the translation. At line 53, the label is moved outside the pie.

The tooltip is added as in the previous example.

The data

Preparing the data has been, again, a problem. Since the Government gives the maps with a code (INE code) and the electoral results with another (alphabetical order), I've had to manipulate the files to merge them, by comparing the municipalities names. Besides, some of the names contain different abbreviations in each file, so they have to be changed by hand...

The files used are:
  • The election results. Is a CSV file with all the municipalities, plus some regions and Barcelona quarters. I have cleaned them so only the municipalities are present. Besides, the file is encoded in Latin1, and the shapefile in UTF-8, so I have converted it using:
    iconv -f latin1 -t utf-8 OPENDATA_A2012_vots.csv > newfile
  • The municipalities shapefile. I have get it from the Vissir3 web site
  • To merge both, I have made a small python script, uploaded to GitHub if you are interested in the code. 
To convert it to TopoJSON, I have run first:

ogr2ogr -simplify 0.001 -f GeoJSON municipis.json  municipis.shp

Simplifying the data so the file is smaller (the number is guessed just by trying many times to get the best size/quality relation)

and later:

topojson -p Name=Name -p ERC=ERC -p CiU=CiU -p PP=PP -p PSC=PSC -p ICV-EUiA=ICV-EUiA -p CUP=CUP -p "C's"="Cs"  -o mun_out_topo.json mun_out.json


To convert JSON to TopoJSON.


Monday, February 4, 2013

Mapping with D3js: Canvas or SVG?

I am using this blog to learn new stuff about mapping. I have been testing Kartograph, and now I'm starting with d3.js.
I wanted to post a first tutorial about mapping with it, but I have found one which, in my opinion,  has everything: http://bost.ocks.org/mike/map/ Besides, the entry links with many good examples.
So I will post how to display a rotating globe, with the two possibilities: SVG and Canvas. I didn't know how to do it, and haven't found a tutorial about that.
As usual, all the code is available at GitHub, where you can find also all the working examples:


Getting the data

In this case, I have used the file used in the d3.js docs. The data must be in the TopoJSON format to work with the example. How to convert a regular GeoJSON into a TopoJSON is also covered in the recommended article.

Basic map using SVG

SVG is an XML vector format, so all the lines and polygons drawn are kept as shapes and can be, therefore, manipulated and styled after their creation.
In this example, a map like the one at the picture above is generated.
var diameter = 320,
    radius = diameter/2;
 
var projection = d3.geo.orthographic()
    .scale(radius - 2)
    .translate([radius, radius])
    .clipAngle(90)
    .rotate([10,0,0]);
 
var svg = d3.select("#map").append("svg")
    .attr("width", diameter)
    .attr("height", diameter);
 
var path = d3.geo.path()
    .projection(projection);
 
 
d3.json("./world-110m.json", function(error, world) {
 
  var globe = {type: "Sphere"};
    svg.append("path")
    .datum(globe)
    .attr("class", "foreground")
    .attr("d", path);
 
  var land = topojson.object(world, world.objects.land);
 
      svg.insert("path")
      .datum(topojson.object(world, world.objects.land))
      .attr("class", "land")
      .attr("d", path);
  
});

  1. The projection variable is set using an ortographic projection, just to see the globe. 
    1. The scale is set from the radius to fit all the globe in the SVG size.
    2. The translate method is a bit tricky. If it's not used, the center of the globe will be at the upper-left corner.
    3. The clipAngle method forces the script to stop drawing at 90 grades. If the angle is not set, the land that is supposed to be hidden is drawn. Just try changing the value.
    4. The rotate method is not necessary, but I have used it to show how it works. The array represents the three axis. So in this case the globe is rotated in the east-west direction.
  2. To create the SVG element, the div with the id=map is selected, and the size is set to the diameter of the globe so the globe fits the space.
  3. The path element is created, with the projection defined before. This is the object that will be used to project the geographic coordinates into the SVG coordinates.
  4. Then, the TopoJSON file is loaded, and only after this is done, the map is drawn.
  5. And here is where the map is drawn. First, the globe is drawn, with a dark border and a blue background that represents the sea. Note that the class foreground is assigned. At the top of the file a style is defined for this class, the same way than in a regular html element.
    Note how a globe is defined, as a sphere type and assigning it with the datum method.
  6. Then the land elements of the TopoJSON are drawn (using the datum method), using land as the class.

Basic map using Canvas

The canvas element is part of HTML5, and allows to draw elements on a raster. It's different from SVG, since once a shape is drawn, there is no way to modify it or style it. And no event handlers on every shape can be set. The memory use of a Canvas is then much lower than the one used by the same drawing using SVG, since there is no DOM to handle.
The Canvas element is better for non-static images, like a rotating globe. This is why most of the map animatinos use the Canvas option.
The map generated in this example is exactly the same than the SVG:
var diameter = 320,
    radius = diameter/2;

var projection = d3.geo.orthographic()
    .scale(radius - 2)
    .translate([radius, radius])
    .clipAngle(90)
    .rotate([10,0,0]);


var canvas = d3.select("#map").append("canvas")
    .attr("width", diameter)
    .attr("height", diameter);

var path = d3.geo.path()
    .projection(projection);

d3.json("./world-110m.json", function(error, world) {
  var land = topojson.object(world, world.objects.land),
      globe = {type: "Sphere"};
      context = canvas.node().getContext("2d");
  
      context.strokeStyle = '#766951';
      
      context.fillStyle = '#d8ffff';
      context.beginPath(), path.context(context)(globe), context.fill(), context.stroke();

      context.fillStyle = '#d7c7ad';
      context.beginPath(), path.context(context)(land), context.fill(), context.stroke();

      
});
The basic parts are more or less the same, but:
  1. Of course, no SVG element is generated, but a canvas one
  2. The stroke and fill colors are set before drawing the elements using strokeStyle and fillStyle. A light blue for the sea and a light brown for the land.
  3. Every time a new kind of element has to be drawn, a path is created with beginPath, and a projection is assigned to it.
  4. The strokes and polygons are drawn using different methods (fill and stroke)

Rotating map using SVG

One of the cool things about D3.js is to make things move. In this example, the projection will be given a rotation to the globe seems to be rotating. The JavaScript part, then must be changed this way:

var diameter = 320,
    radius = diameter/2,
    velocity = .01,
    then = Date.now();

var projection = d3.geo.orthographic()
    .scale(radius - 2)
    .translate([radius, radius])
    .clipAngle(90);

var svg = d3.select("#map").append("svg")
    .attr("width", diameter)
    .attr("height", diameter);

var path = d3.geo.path()
    .projection(projection);

var globe = {type: "Sphere"};
svg.append("path")
    .datum(globe)
    .attr("class", "foreground")
    .attr("d", path);


d3.json("./world-110m.json", function(error, world) {

  var land = topojson.object(world, world.objects.land),
      globe = {type: "Sphere"};

      svg.insert("path", ".graticule")
      .datum(topojson.object(world, world.objects.land))
      .attr("class", "land")
      .attr("d", path);

  d3.timer(function() {
    var angle = velocity * (Date.now() - then);
    projection.rotate([angle,0,0]);
    svg.selectAll("path")
      .attr("d", path.projection(projection));
  });
  
});

  1. The SVG paths are inserted the same way, but a timer is set after appending the land zones (the globe stays with the same shape, so no rotatino is needed)
  2. The timer changes the angle in the step defined in the velocity variable, and tht projection is changed to this angle.
  3. The path is modified with the path that has the new projection.
Using SVG, no path has to be re-drawn.

Rotating map using Canvas

The main difference in the example using Canvas is that all the picture has to be drawn with each rotation:
var diameter = 320,
    radius = diameter/2,
    velocity = .01,
    then = Date.now();

var projection = d3.geo.orthographic()
    .scale(radius - 2)
    .translate([radius, radius])
    .clipAngle(90);

var canvas = d3.select("#map").append("canvas")
    .attr("width", diameter)
    .attr("height", diameter);

var path = d3.geo.path()
    .projection(projection);

d3.json("./world-110m.json", function(error, world) {
  var land = topojson.object(world, world.objects.land),
      globe = {type: "Sphere"};

  
  d3.timer(function() {
    var angle = velocity * (Date.now() - then);
    projection.rotate([angle,0,0]);

    context = canvas.node().getContext("2d");
    context.clearRect(0, 0, diameter, diameter);

    context.strokeStyle = '#766951';

    context.fillStyle = '#d8ffff';
    context.beginPath(), path.context(context)(globe), context.fill();
    context.beginPath(), path.context(context)(globe), context.stroke();

    context.fillStyle = '#d7c7ad';
    context.beginPath(), path.context(context)(land), context.fill();
    context.beginPath(), path.context(context)(land), context.stroke();

  });
});
  1. After loading the land zones, the timer is set.
  2. In each step, the angle is calculated like in the SVG, but then, all the Canvas is cleared with the method clearRect.
  3. The map is re-drawn again like in the static example.

Tuesday, January 15, 2013

Kartograph tutorial III: Symbols


Adding symbols to a map is almost mandatory. Kartograph has this capability using the addSymbols method.
By the way, the documentation page is not well linked in the web.  The right symbols documentation page is here.

The data

As usual, all the data used in the example is available. I have switched to GitHub, so the files are there.I have also created a web page to put all the stuff related to the blog: http://rveciana.github.com/geoexamples/
We will use a map of Morocco for the examples. All the borders and cities are taken from Natural Earth. The pages are at the "1:10m cultural vectors" page, and the layers used are countries and populated places.

Since the web page will need an SVG image to get the map data, generate using the command:

    kartograph -o morocco.svg morocco.json

Then, we will use a csv file to read the data we will put in the map, so ogr2ogr is necessary to convert the files from the original shapefile:

ogr2ogr -clipdst -12 26 0 38 -f csv csv ne_10m_populated_places.shp

Note that the option -clip is used to reduce the output file size, which is quite critical when drawing it with JavaScript.

Drawing the data as bubbles


The basic way to draw point data using Kartograph (or the most used in the examples) is what they call bubbles. A bubble is a circle with a user defined radius that may be used to represent data values.
Let's see the Javascript corresponding to this working example:
function loadMap(){
var map = Kartograph.map('#map', 500, 0);
map.loadMap('morocco.svg', function() {
     
    map.addLayer('world',{'name':'bg',
                'styles': {
                    'stroke': '#aaa',
                    'fill': '#f6f4f2'
                }});
    map.addLayer('world',{'name':'moroccobg'});
    map.addLayer('world',{'name':'moroccofg'});
 
     
    map.getLayer('moroccobg').style('stroke', function(data) {
       return data.country == "Morocco" ? '#d8d6d4' : '#aaa';
    }).style('stroke-width', function(data) {
       return data.country == "Morocco" ? 10 : 1;
    }).style('stroke-linejoin', function(data) {
       return data.country == "Morocco" ? 'round' : null;
    });
     
    map.getLayer('moroccofg').style('stroke', function(data) {
       return data.country == "Morocco" ? '#333' : '#aaa';
    }).style('fill', function(data) {
       return data.country == "Morocco" ? '#fff' : null;
    });

    $.ajax("./data/csv/ne_10m_populated_places.csv").done(
     function(data) {
        var points = [];
        var textLines = data.split(/\r\n|\n/);
        for (var i=1; i<textLines.length; i++) {
            var fields = textLines[i].split(',');
            points.push({lon: parseFloat(fields[22]), lat: parseFloat(fields[21]), name: fields[4], population: parseInt(fields[26])});
        }
         
        $.fn.qtip.defaults.style.classes = 'qtip-light';
        var scale = $K.scale.sqrt(points, 'population').range([0, 40]);

        map.addSymbols({
            type: Kartograph.Bubble,
            data: points,
            location: function(d) { return [d.lon, d.lat]; },
            radius: function(d) { return scale(d.population);},
            tooltip: function(d) {
                return '

'+d.name+'

population: '+d.population; }, sortBy: 'radius desc', style: 'fill:#800; stroke: #fff; fill-opacity: 0.4;' }); }); }, { padding: -2 }) };

Let's see the different parts of this first example:
  •  First the background data is loaded into a map object. Lines 2-3
  • Then, the layers are added with the addLayer method, using alias so the same layer can get more than one style, as explained here
  • After that, the styles are set to each layer, up to the line 26
  • Then, using the jQuery ajax function, the csv file is retrieved. This file contains de big cities location, name and population, that will be used to draw the bubbles.
  • Once the csv file is loaded, the code can draw the bubbles. First, the csv is parsed into an array points containing objects with the necessary info. Lines 30-35.
  • The qtip is inicializad. This will enable the tooltips showing the city name when the mouse is over the bubble. Line 37
  • A scale is defined. This Kartograph function takes all the data (population field from the points array), gets the maximum and minimum values, applies the function we want to set the radius (sqrt, so the circle area is proportional to the population). Then, the range method sets the minimum and maximum radius we want to use. Line 38
  • Then, the bubbles are added using the addSymbols method. In this case, the options used are: Line 40
    • type: The Bubbles type is set here. I haven't found a good list of all the available types.
    • data: The points array.
    • location: Where in every item in the data array the longitude and latitude can be found. In this case, the data is passed in an array, but the object can be also a kartograph.LonLat object.
    • radius: In this case, the scale function defined before is called, but any other possibility is OK if the function returns the number of pixels. The radius can be a fixed number too.
    • tooltip: When filled, this option sets the tool tip shown when the mouse is over the symbol. To do so, the jquery.qtip.js must be included, although is used with this option instead what you will find in the official qtip docs. Any html code can be added here. 
    • order: Sets the order to draw the symbols. Ordering by radius allows the small icons to be over the big ones, so the 
    • style: Finally, the bubble style can be set here
  • The padding is set as the second parameter of the loadMap method.  This clips a little the map, so it the lines closing the polygons are not visible. Check which value is better in every case. Line 56

Clustering 

 

As you can see in the previous example,  if the number of bubbles to draw in a small area is too big, the result is not very attractive. To avoid that, two clustering options are provided:
  • Noverlap: Avoids, as the name suggests, the bubbles overlapping.  A small overlapping can be allowed using the tolerance parameter
  • k-Means: First, a number of bubbles is defined (64 by default). Then, every element is assigned to one of the clusters.
You can compare the results and see the different source codes at this web page. Let's see only the part of the code that changes so the post keeps a reasonable length. Using the noverlap method as in this example web:

map.addSymbols({
            type: Kartograph.Bubble,
            data: points,
            location: function(d) { return [d.lon, d.lat]; },
            radius: function(d) { return scale(d.population);},
            tooltip: function(d) {
                return '

'+d.name+'

population: '+d.population; }, clustering: 'noverlap', aggregate: function(d){ var name = "" var population = 0; $.each(d, function(i, city) { name += " "+city.name; population += city.population; }); return { name: name, population: population }; }, sortBy: 'radius desc', style: 'fill:#800; stroke: #fff; fill-opacity: 0.4;' });

  • clustering: This option appears so be set as noverlap or k-mean. Line 11
  • aggregate: This is the important part. The structure returned here will overwrite the original from the data. The position is already calculated, but the other parameters can be re-calculated.
    In our case, since several cities are added as one single bubble, the most logical is to represent this point with the aggregate population. The parameter d in the function will be an array with all the points, so we use the .each method from jQuery to iterate. Line 15.
    The name is also overwritten to show all the cities included in the new bubble.
If the clustering option is using k-Means, the clustering option will be changed as follows:
clustering: 'k-means',
clusteringOpts: {
    size: 64
},

Changing the size tag to the number of clusters to represent. I think that in this case is important to tune, since the results change a lot.

As a final comment to the Kartograph clustering, there is a thing I don't understand: Some bubbles present when no clustering is applied disappear with the noverlap method. For instance, look at the comparison example to the city of Tindouf, the most southern of all. The city disappears in the noverlap method, and is not aggregated to any new bubble, although no overlap occurs. When using the k-Means, is properly aggregated to other bubbles.

 

Image icons

Is possible to use images as symbols instead of bubbles. This allows to put fixed icons as in the cool La Bella Italia example.
Here, we will see how to represent the geotagged Flickr images on a map (which was the original idea for this post).
There are two differences from the original code.
The first one is that Flickr offers feeds in JSON format. The cool thing is that is possible to get only pictures from a selected country, region or city. So the example could be used at city level.
The code and the working page are here, but the part that changes is, of course the parsing of the csv file, which looks now like this:

$.getJSON("http://www.flickr.com/services/feeds/geo/Morocco?jsoncallback=?",
     {
       format: "json"
     },
     function(data) {
        var points = []
        $.each(data.items, function(i,item){
            points.push([new $K.LonLat(item.longitude,item.latitude),item.title,item.media.m]);
            if ( i == 100 ) return false;
        });

Is much easier than the other, since the JSON file is transformed directly into an object.
The second difference is the addSymbols:
map.addSymbols({
    type: $K.Icon,
    data: points,
    location: function(d) {return d[0]; },
    icon: function(d) { return d[2]; },
    iconsize: [30,30],
    offset: [-15,-15],
    title: function(d) {return d[1]; },
    tooltip: function(d) {return [d[1],''];},
});

Note that:
  • Here, the LonLat object has been used instead of the array to set the location (just to show how to do it).
  •  The type is now Icon
  • The label icon sets the image source, which is part of the Flickr feed in this example.
  • The iconsize label can change the icon size. By default is 10x10 pixels.
  • The offset label centers the icon (I have checked that the location is set at the left top of tie image)
  • Tooltip and title must be set this way to give the result shown. Both interact with the qtip plugin.
Finally, I'm very frustrated with the clustering when using icons. In this case is completly necessary, since many pictures are at the same touristic points, but I haven't found how to do it. Maybe somebody can help.

Monday, December 10, 2012

Raster calculations with GDAL and numpy: calculating GFS wid speed

Performing raster calculations is a frequent need. GDAL python bindings come with a utility program,  gdal_calc.py, that does that, but many times is useful to code them.
In this example, we will be able to calculate the wind speed field from the GFS model, from the given u and v speed.
 The calculated wind field. America is easily seen at the right side of the image

The data

In this example, we will use the GFS global model data, which is available at their web site. Just navigate to a date that pleases you and download a .grb file.
Files are in grib format, with a little problem when trying to view them in QGis: Coordinates are from 0 to 360º in longitude, while the usual thing is to put them from -180º to 180º. Everything will work without changing this, but you can do it after taking a look to this post.

Inside the grib file, there are many bands (hundreds). You can see their definition using the gdal command gdalifo, or the file with the extension *inv at the same place you downloaded the grib file. There are many wind fields, that correspond to different altitudes, all them indicated as u (east) and v components (north). Just take note of the band number of both at the desired level.

NumPy

The best of using the python gdal bindings, compared from those of the other programs, is that converting a raster file to a NumPy matrix is now very easy. 
NumPy is a python extension that allows to perform matrix calculations in a really easy and efficient way. It should be installed in your computer if GDAL python is installed.
Once you have two matrices A and B in NumPy, adding element by element to a C matrix would be as easy as:
C = A + B
and multiplying:
C = A * B
Note that is not a matrix multiplication (more info here), but this is what we usually need when working with raster images.

The code

The wind speed is the modulus of the two components, so 
spd = sqrt(u^2+v^2)
The "traditional way" to calculate it would be opening two bands and loop all the elements to create the new band. Instead, this can be done:
from osgeo import gdal
from osgeo.gdalnumeric import *
from osgeo.gdalconst import *

fileName = "gfs.tiff"
bandNum1 = 199
bandNum2 = 200

outFile = "out.tiff"

#Open the dataset
ds1 = gdal.Open(fileName, GA_ReadOnly )
band1 = ds1.GetRasterBand(bandNum1)
band2 = ds1.GetRasterBand(bandNum2)

#Read the data into numpy arrays
data1 = BandReadAsArray(band1)
data2 = BandReadAsArray(band2)

#The actual calculation
dataOut = numpy.sqrt(data1*data1+data2*data2)

#Write the out file
driver = gdal.GetDriverByName("GTiff")
dsOut = driver.Create("out.tiff", ds1.RasterXSize, ds1.RasterYSize, 1, band1.DataType)
CopyDatasetInfo(ds1,dsOut)
bandOut=dsOut.GetRasterBand(1)
BandWriteArray(bandOut, dataOut)

#Close the datasets
band1 = None
band2 = None
ds1 = None
bandOut = None
dsOut = None

Note that:
  1. gdalnumeric is used, because it's the easiest way to dump a raster into a NumPy matrix using BandReadAsArray and the inverse function with BandWriteArray.
  2. In this case, we are reading the bands 199 and 200, but you should check which bands are the ones to read in the file you download.
  3. CopyDatasetInfo will copy all the input file metadata into the output file. This includes the geotransform, the projection, the GCPs (there aren't in this case) and the other metadata.
  4. Don't forget to set the bands and datasets to None at the end to free them. In longer scripts this may be critical.

gdal_calc.py

If you want to use the command line function to do the same, you can try with:
gdal_calc.py -A gfs.tiff --A_band 199 -B gfs.tiff --B_band 200 --outfile out.tiff --calc="sqrt(A*A+B*B)"
It gives the same result.

Thursday, November 29, 2012

kartograph tutorial II: going interactive

In my last post I showed how to create a cool SVG map using Kartograph, to be opened and edited with an Inkscape-like program.
The elections were celebrated and CiU won again, but with less votes. I could either update the map or play with the same data. The fun with Kartograph is using the generated SVG to create interactive web pages using Kartograph.js, so this is what I'll do.
The web page will be able to:
  • Generate the map not only for the first position, but also for the others
  • Query every municipality to know which were the results there.
As usual, the code is available to download, and at the working web page too.

This second example has a continuation post: Kartograph tutorial III: Symbols
 
Changes to the SVG
In the other post, we used the SVG only to draw, but now we need to add data to it, so we can query it.
So change the municipalities layer at the file elections.json from:
   "municipalities":{
       "src": "./mun_out.shp"
   }

to:
   "municipalities":{
       "src": "./mun_out.shp",
       "attributes": {"winner": "Winner","ciu": "CiU","psc":"PSC-PSOE","erc":"ERC","pp":"PP","icv":"ICV-EUiA","cs":"C's","name":"Name"}
   }

This will add the attributes set in the shapefile geometries to the SVG geometries.
To generate the SVG again, just type:
kartograph -o elections.svg elections.json

Viewing the map in a web page

Even though is quite simple, I haven't found any really basic working example, so here it is:

<html>
    <head>
        <script src="jquery-1.8.3.min.js"></script>
        <script src="raphael-min.js"></script>
        <script src="kartograph.min.js"></script>
 <script language="JavaScript">
function loadMap(){
var map = Kartograph.map('#map', 600, 0);

map.loadMap('elections.svg', function() {

    map.addLayer('background');
    map.addLayer('world');
    map.addLayer('depth');
    map.addLayer('trees');
    map.addLayer('crops');
    map.addLayer('municipalities');
     
}, { padding: -30 })
}
</script>
    </head>
    <body onLoad="loadMap()">

        <div id="map"></div>
    </body>
</html>

  1. Include jquery, raphael and kartograph
  2. The map must be created when the page loads, so the function loadMap() is called at the body tag , using the onLoad event.
  3. Kartograph.map('#map', 600, 0) creates the map at the div tag with the id map. X size will be 600 px, and the aspect ratio will be respected, because the y size is 0. When an other y is set, strange things happen with the png borders.
  4. loadMap draws the map from the file elections.svg. Then, executes a function. In this case, the function loads all the layers one by one, using the method addLayer(LayerNameInTheSVG)
  5. padding: -30 removes the strange lines from the vectors in the borders by padding the image. Try without using it to understand what happens.


This renders the SVG without any styling or interaction. But is the basic way to start any Kartograph.js project.

CSS Styling

There are two ways to add styles to the map. We will use both, to show them.
The first one, is to use a separate css file, as in the last post.
Adding the css file is quite tricky, because of the IExplorer compatibility. Now, the loadMap call would look like:

map.loadMap('elections.svg', function() {
 map.loadCSS('elections.css', function() {
  map.addLayer('world');
   ...
 });
})
So, the only thing that the loadMap method does, is calling the loadCss method, which will do all the stuff.

The css file is also different than the one used before. To style the layer world, the css section will be:
#map svg path.world {
stroke: #fff;
fill: #cccccf;
fill-opacity: .7;
}

Tip1: Be careful, Chrome doesn't reload your css even though you use ctrl-F5!
Tip2: Do not separate the CSS elements with comas (look at the example). With the python example, this didn't matter.
The other way is using the style method from the layer, as we will see in the final part of the post.

Adding tooltips

If we want the web page to show a tooltip with the result when the mouse is over a municipality, just add a tooltip to the desired layer (municipalities, in this case). The function in loadCSS will be now:
map.loadCSS('elections.css', function() {
 ...
 map.addLayer('municipalities');

$.fn.qtip.defaults.style.classes = 'qtip-light';

 map.getLayer('municipalities').tooltips(function(data) {

return [data.name, 'CiU: <b>' + data.ciu + '</b><br/>PSC-PSOE: <b>'+ data.psc + '</b><br/>ERC: <b>' + data.erc+'</b><br/>PP: <b>'+data.pp+'</b><br/>ICV-EUA: <b>'+data.icv+'</b><br/>C\'s: <b>'+data.cs+'</b>'];
});
});

A function is called as a parameter to the tooltips method of the layer municipalities. This function returns what we want to be shown in the tooltip. The data parameter has all the attributes in the SVG geometries. In this case, a list with the results is shown.
The other line in the code just sets the style of the tooltip. The official documentation has more information about the options.
Also, the tooltips styles and javascript must be included in the header, adding:

<script src="jquery.qtip.js"></script>
 <link rel="stylesheet" type="text/css" href="jquery.qtip.css"/>


Setting the colors in the map

Finally, the map will be colored. By default, depending on the winner party, but with a combo box to choose the position.
So first, a function is created to sort the results in every municipality, and set the style with a different color for every party:

function setColors(value){
 map.getLayer('municipalities').style('fill', function(data) {

    var positions = new Array();
    positions[0] = parseInt(data.ciu);
    positions[1] = parseInt(data.psc);
    positions[2] = parseInt(data.erc);
    positions[3] = parseInt(data.pp);
    positions[4] = parseInt(data.icv);
    positions[5] = parseInt(data.cs);

    positions.sort(function(a,b) { return b-a; });

    var bgColor;
    if (positions[value-1] == parseInt(data.ciu)){
        bgColor = "#99edff";
    } else if (positions[value-1] == parseInt(data.psc)){
        bgColor = "#ff9999";
    } else if (positions[value-1] == parseInt(data.erc)){
        bgColor = "#EDE61A";
    } else if (positions[value-1] == parseInt(data.pp)){
        bgColor = "#005aff";
    } else if (positions[value-1] == parseInt(data.icv)){
        bgColor = "#74ff74";
    } else if (positions[value-1] == parseInt(data.cs)){
        bgColor = "#ff7b00";
    }
    
    
    return bgColor;
 });
}
First, we set the style to the layer with the method style. the first argument is the property to change, fill in this case. The second parameter is the value to set. In this case, it will depend of the function that chooses the position. Things to look at:
  • value is the position to show (from 1 to 5)
  • To sort the positions in the array in descending order, use the sort() method, with a function to order the elements as numbers. See why here.
  • Then, check which party corresponds to the number of votes at the desired position, and set the color.
This function will be called at the beginning to show the first position by default:
setColors(1);
and a select element is created to select the position and change the colors when the option is changed:
<select onChange="changePos(this)">

<option value="1">1st position</option>

<option value="2">2nd position</option>
<option value="3">3rd position</option>
<option value="4">4th position</option>
<option value="5">5th position</option>
</select>
An auxiliar function is created to get the option value:
function changePos(sel){
 var value = sel.options[sel.selectedIndex].value;
 setColors(value);
}
 
That's all!
The best is to download the code or look at the working web.
Of course, some styling is still needed to do a deliverable web page, but the main idea is already done.