d3-composite-projections

Some countries have regions sparse around the globe, which adds difficulties when drawing maps for them.

D3 already had the albersUsa projection that solved this problem by creating a composed projection, moving Alaska and Hawaii close to the main part of the USA.

But the other countries didn't have a projectino like this. That's why I made this library.

It adds the composite projection for:

• Spain
• France
• Portugal
• USA

With a function that draws a border between the composition zones by returning an SVG path.

There is an example for each region, linked in the list above.

The library web page explains the usage and  installation/testing

If you are going to use it, need more regions/countries or find an error, please leave a comment here.

D3js mapping presentation at Girona

Every year, SIGTE organizes workshops and a conference about Free GIS software in Girona.

This year I gave a workshop about D3js mapping.
The slides (in Spanish) are here: http://rveciana.github.io/Mapas-web-interactivos-con-D3js/
The examples can be found at my bl.ocks.org space: http://bl.ocks.org/rveciana named with the prefix JSL 2015.

Using the D3 trail layout to draw the Hayian tracks

I wrote many examples (1, 2, 3 and 4) and some entries in the blog (1 and 2) showing how to draw animated paths on a map using the D3 library.
But since then, Benjamin Schmidt wrote a D3 layout, called trail layout, that simplifies a lot doing this kind of stuff.
Since the layout is new, and hasn't got many examples (actually, two made by the author), I'll try to show how to work with it.

The trail layout

How does the trail layout work? The author defines it as:

This is a layout function for creating paths in D3 where (unlike the native d3.svg.line() element) you need to apply specific aesthetics to each element of the line.

Basically, the input is a set of points, and the layout takes them and creates separate segments to join them. This segments can be either line or d SVG elements.

Let's see the simplest example:

• In this case, the points are defined as an array of objects with the x and y properties. If the x and y are named this way, the layout takes them directly. If they are called for instance lon and lat, the layout must be told how to get them.
• Line 10 creates the SVG
• Line 14 initializes the layout. In this case, the layout is using the coordType xy, which means that as a result will give the initial and end point for each segment, convenient for drawing SVG line elements. The other option is using the coordinates value, which is convenient for drawing d elements, as we will see later.
•  Line 15 is where the data is set and the layout is retrieved
• The last step is where the lines are actually drawn. 
• For each data element, the line svg is added
• The styles are applied
• The extremes of the line are set using the attributes x1, y1, x2, y2

How to use coordinates as the coordType:

The following example created the trail as a set of SVG line elements, but the trail layout has an option for creating it as a set of SVG d elements (paths).
You can see the example here. The data, in this case, is the Hayian track. As you can see, it's quite similar as the former example, with the following differences:

• Since in this case we are using geographical coordinates, a projection must be set, and also a d3.geo.path to convert the data into x and y positions, as usual when drawing d3 maps
• When initializing the trail layout, coordinates must be set as the coordType.
• Since the data elements do not store the positions with the name x and y, the layout has to be told how the retrieve them using the positioner:
  .positioner(function(d) {return [d.lon, d.lat];})
• When drawing the trail, a path element is appended instead the line element, and the d attribute is set with the path  function defined above.

Creating the map with the trail

 

Once the basic usage of the trail layout is known, let's reproduce the Hayian path example (simplified for better understanding):

• The map creation is as usual (explained here)
• Lines 49 to 51 create the trail layout as in the former example
• Line 67 creates the trail, but with some differences:
• the beginning and the end of the line are the same point at the beginning, so the line is not drawn at this moment (lines 69 to 72)
• The stroke colour is defined as a function of the typhoon class using the colour scale (line 75)
• A transition is defined to create the effect of the line drawing slowly
• The ease is defined as linear, important in this case where we join a transition for each segment.
• The delay is set to draw one segment after the other. The time (500 ms) must be the same as the one set at duration
• Finally, the changed values are x2 and y2, that is, the final point of the line, which are changed to their actual values
• The complete example, with the typhoon icon and the date is also available

It's possible to use paths instead of lines to draw the map, as in the first version. The whole code is here, but the main changes are in the last section:

hayan_trail.enter()
      .append('path')
      .attr("d", path)
      .style("stroke-width",7)
      .attr("stroke", function(d){return color_scale(d.class);})
      .style('stroke-dasharray', function(d) {
        var node = d3.select(this).node();
        if (node.hasAttribute("d")){
          var l = d3.select(this).node().getTotalLength();
          return l + 'px, ' + l + 'px';
        }
      })
      .style('stroke-dashoffset', function(d) {
        var node = d3.select(this).node();
        if (node.hasAttribute("d"))
          return d3.select(this).node().getTotalLength() + 'px';
      })
      .transition()
      .delay(function(d,i) {return i*1000})
      .duration(1000)
      .ease("linear")
      .style('stroke-dashoffset', function(d) {
          return '0px';
      });

• The strategy here is to change the stroke-dasharray  and stroke-dashoffset style values as in this example, and changing it later so the effect takes place.
• At the beginning, both values are the same length as the path. This way, the path doesn't appear. The length is calculated using the JavaScript function getTotalLength
• After the transition, the stroke-offset value will be 0, and the path is fully drawn

Conclusion

I recommend using the trail layout instead of the method from my old posts. It's much cleaner, fast, easy, and let's changing each segment separately.

The only problem I find is that when the stroke width gets thicker, the angles of every segment make strange effects, because the union between them doesn't exist. 

This didn't happen with the old method. I can't imagine how to avoid this using lines, but using the coordinates option could be solved transforming the straight lines for curved lines.

Links

The first simple example
The simple example using coordinates
Third example: The animated trail
The complete example with the icon and date
The complete example using coordinates

The trail layout page

The original example of a path with gradient, without the trail layout

D3 map Styling tutorial IV: Drawing gradient paths

After creating the last D3js example, I was unsatisfied with the color of the path. It changed with the typhoon class at every moment, but it wasn't possible to see the class at every position. When I saw this example by Mike Bostock, I found the solution.

Understanding the gradient along a stroke example

First, how to adapt the Mike Bostock's Gradient Along Stroke example to a map.

The map is drawn using the example Simple path on a map, from this post. The only change is that the dashed path is changed with the gradient.
You can see the result here.

The differences from drawing a simple path start at the line 100:

var line = d3.svg.line()
      .interpolate("cardinal")
      .x(function(d) { return projection([d.lon, d.lat])[0]; })
      .y(function(d) { return projection([d.lon, d.lat])[1]; });

  svg.selectAll("path")
      .data(quad(sample(line(track), 8)))
    .enter().append("path")
      .style("fill", function(d) { return color(d.t); })
      .style("stroke", function(d) { return color(d.t); })
      .attr("d", function(d) { return lineJoin(d[0], d[1], d[2], d[3], trackWidth); });

•  The line definition remains the same. From every element it gets, it takes the lat and lon attributes, projecting them, and assigning them to the x and y path properties
• A color function is defined at line 41, which will interpolate the color value from green to red:
  var color = d3.interpolateLab("#008000", "#c83a22");
• The data is not the line(track) directly, as in the former example, but passed through the functinos sample and quad.
• The sample function assigns a property t with values between 0 and 1, which is used to get the color at every point.
• Finally, the function lineJoin is used to draw a polygon for the sampled area.

The functions used in the Mike Bostock's example aren't explained, I'll try to do it a little:

• sample takes a line (the data applied to a line function), and iterates with the precision parameter as increment along the string, creating an array with all the calculated points.
• quad takes the points calculated by the sample function and returns an array with the adjacent points (i-1, i, i+1, i+2).
• lineJoin takes the four points generated by quad, and draws the polygon, with the help of lineItersect and perp functions.

Drawing the typhoon track with the colors according to the typhoon class

The final example draws the typhoon path changing smoothly the color according to the typhoon class.
The animation of the path, and the rotating icon are explained in the third part of the tutorial. In this case, the way to animate the path will change.
For each position of the typhoon, a gradient path is drawn, because the gradient is always between two colors. So the part of the code that changes is:

      //Draw the path, only when i > 0 in otder to have two points
      if (i>0){
        color0 = color_scale(track[i-1].class);
        color1 = color_scale(track[i].class);

        var activatedTrack = new Array();
        
        activatedTrack.push(track[i-1]);
        activatedTrack.push(track[i]);

        var color = d3.interpolateLab(color0, color1);
        path_g.selectAll("path"+i)
        .data(quad(sample(line(activatedTrack), 1)))
        .enter().append("path")
          .style("fill", function(d) { return color(d.t);})
          .style("stroke", function(d) { return color(d.t); })
          .attr("d", function(d) { return lineJoin(d[0], d[1], d[2], d[3], trackWidth); });
      }

      i = i + 1;
          if (i==track.length)
            clearInterval(animation)

• Inside the animation interval (line 145), the gradient path is create for each position (starting with the second one to have two points)
• The two colors are taken from the point information
• An array with the two points is created, with the name activatedTrack. I tried using more points, but the result is very similar.
• The color interpolation is calculated (line 172)
• The gradient colored path is created (line 173). Note that the name is path+i, to make different paths each iteration, and not to overwrite them. The method is the same as the one used in the first section.

Besides, an invisible path with all the positions is created, so the typhoon icon can be moved as it was in the third part of the tutorial.

Links

• The final example 
• A simple gradient along the path
• D3 map Styling tutorial I: Preparing the data
• D3 map Styling tutorial II: Giving style to the base map
• D3 map styling tutorial III: Drawing animated paths
• Mike Bostock's Gradient Along Stroke example

D3 map Styling tutorial II: Giving style to the base map

The example La Belle France, or the original La Bella Italia by Gregor Aisch use svg filters to give style to the maps. I haven't found many examples about how to do it, so I will explain here two different styles: a simple shadow under the map, and a style similar to the original map.

As usual, the examples can be seen at my bl.ocks.org page:
Drop shadow
LaBella Italia

SVG filters basics

SVG can style the elements by using css or css-like attributes, like the color, the stroke, and so on. But it's possible to use filters to add efects such as blurring, dilating the shapes, adding lights... This, of course, can be used to change the map polygons too.

A basic usage of an SVG filter is like this:

And the result is:

Note that:

• The shape (a rectangle defined by the rect tag) is drawn as usual
• The filter is defined with the filter tag inside the defs section. The filter must have an id to be used in the geometries where it has to be used
• The filter is applied to the rectangle by using the filter="url(#blur)" attribute, where the id of the filter is added as url(#name_of_the_filter)

This is the easiest filter an SVG can have, but filters can concatenate different effects one after the other or in parallel, as we will see in the following examples. 

A list of all the effects and a tutorial can be found at W3C Schools

Simple shadow

The drop shadow effect gives a quite nice look to the maps and it's fast to process by the browser and easy to code. I took the code from this example.

The map shown in the last post with the effect. The code:

Note that the filter (lines 46 to 66) has several parts:

• The filter is defined adding a defs tag and appending the filter, with an id and a height tags. The height is added to affect the region outside the geometry. It's not actually necessary in this case, but I kept it to maintain the example code.
• An feGaussianBlur is then appended (line 50). The in attribute indicates that the input image for the filter is the alpha value of the input image, so if the map is colored in red, the shadow will remain grey. stdDeviation indicates the filter intensity, and output gives an id to use the resulting image, as we will see.
• Line 55 adds an feOffset filter, that will move the shadow. The dx and dy attributes define how to move the shadow, but the important attribute here is the in, that takes the id defined in the feGaussianBlur tag, so what is moved is the blurred image. Again, an out attribute is defined to use the result image later.
• Line 61 defines an feMerge tag. This is an interesting property of svg filters that allow to append different filter results one over the other to create more complex outputs. Appended to the tag, an feMergeNode is added at line 63 with the blurred image at the in attribute. At line 65, the original image is appended over the blurred image, with an other feMergeNode tag, with the in attr set to SourceGraphic to indicate the original image.

SVG filters seemed quite dificult to me, but they are actually a kind of instructions one after the other, written in XML.

Styling like La Bella Italia

As it was shown in the post, the filters part of La Bella Italia map is quite complicated. I'll try to show an easy way to get a similar effect. 

Disclaimer: In my old computer, generating these maps is very slow, so maybe it would be better to think about another combination to create cool maps without using the erode and dilate filters.

The map without any effect looks like this:

And the effect has three stacked parts (I'll paste only the part of the code affecting each part, the whole code is here):

1. A colouring and blurring
   
   To do it:
   

   filter.append("feColorMatrix")
       .attr("in","SourceGraphic")
       .attr("type", "matrix")
       .attr("values", "0 0 0 0 0.6 0 0 0 0 0.5333333333333333 0 0 0 0 0.5333333333333333  0 0 0 1 0")
       .attr("result","f1coloredMask");
   filter.append("feGaussianBlur")
     .attr("in", "f1coloredMask")
     .attr("stdDeviation", 15)
     .attr("result", "f1blur");

   So there are two concatenated actions. The first one, feColorMatrix, changes the color of the original image (see how here). The second, blurs it as in the first example.
2. Eroding, coloring, blurring and composing:
   
   

   filter.append("feColorMatrix")
       .attr("in","SourceGraphic")
       .attr("type", "matrix")
       .attr("values", "0 0 0 0 0   0 0 0 0 0   0 0 0 0 0   0 0 0 500 0")
       .attr("result","f2mask");
   filter.append("feMorphology")
       .attr("in","f2mask")
       .attr("radius","1")
       .attr("operator","erode")
       .attr("result","f2r1");
   filter.append("feGaussianBlur")
       .attr("in","f2r1")
       .attr("stdDeviation","4")
       .attr("result","f2r2");
   filter.append("feColorMatrix")
       .attr("in","f2r2")
       .attr("type", "matrix")
       .attr("values", "1 0 0 0 0.5803921568627451 0 1 0 0 0.3607843137254902 0 0 1 0 0.10588235294117647 0 0 0 -1 1")
       .attr("result","f2r3");
   filter.append("feComposite")
       .attr("operator","in")
       .attr("in","f2r3")
       .attr("in2","f2mask")
       .attr("result","f2comp");

   This one is more complicated. 
1. The first step changes the map color into black, with an alpha value of 0.5. This output will be used in the second and last step.
2. Then, using the first output, the image is eroded. That is, the land gets smaller by one pixel, using feMorphology with the erode operator. Then, the result is blurred using feGaussianBlur, and coloured as in the first filter.
3. The resulting image is composited with the original black and white image, using feComposite. The definition of this operator, taken from here, is:
   

   The result is the part of A that is within the boundaries of B. Don't confuse the name of this attribute value with the in attribute.

3. The two effects are stacked, and the original map is added at the end:
   
   

   var feMerge = filter.append("feMerge");
   
   feMerge.append("feMergeNode")
       .attr("in", "f1blur");
   feMerge.append("feMergeNode")
       .attr("in", "f2comp");
   feMerge.append("feMergeNode")
       .attr("in", "SourceGraphic");

   This part is simple,the feMerge stacks all the outputs indicated in the feMergeNode tags. So in this case, the blurred image goes in the first place, then the eroded one, and finally, the original one. Now, the effect seems quite a lot to the one in La Bella Italia.

Links

Drop shadow map - The first example
La Bella Italia like map - The second example
D3 map Styling tutorial I: Preparing the data - First part of the tutorial
La Bella Italia - Kartograph example by Gregor Aisch
La Belle France - The same example as La Bella Italia, but using D3js
d3.js drop shadow example - SVG drop shadow filter by Charl P. Botha
SVG filters tutorial - W3C Schools tutorial about SVG filters

D3 map Styling tutorial I: Preparing the data

This is the first post of a series to explain how to draw styled maps with paths in them, using D3js.
The final map will be an animated map of the Haiyan typhoon track:

This entries are to explain my last post La Belle France, which had a part with filters and an other with paths, not very clearly explained.
So first, the typhoon path data is needed, and a base map to draw under it too.

Base Map

To draw the maps for this tutorial, the wolrd map at 50m resolution used in most of the mbostock's examples will be used. The base map, is taken from this example, and can be seen, as usual, at bl.ocks.org.

The basic difference from the Mercator example is the projection variable:

var projection = d3.geo.mercator()
    .scale(5*(width + 1) / 2 / Math.PI)
    .translate([width / 2, height / 2])
    .rotate([-125, -15, 0])
    .precision(.1);

• The scale is multoplied by five, to zoom to the typhoon zone
• A rotation is added, to center the map to the typhoon zone

Typhoon Haiyan track

It's not easy to find the actual data for the hurricanes and typhoons in a convenient format. After looking into several sites, the one I like the most is this one from Japan. The data is in a list which is possible to copy, but putting all the points into an array is not an amusing activity. So here is the script I've used to transform the data into a json:

f = open("track.txt","r")
out = "["
for line in f:
    day = int(line[12:16])
    hour = int(line[16:18])
    lat = float(line[20:24])
    lon = float(line[28:34])
    h_class = int(line[48:49])

    out += '{"day":%d, "hour":%d, "lat":%.1f, "lon":%.1f, "class": %d},' %(
        day, hour, lat, lon, h_class)
out = out[0:-1] + "]"
f.close()

print out

Where track.txt is the file where I pasted the data copied from the web page.

Links

The base map for the next posts
La Belle France - The blog entry that made me write this post
D3 Mercator world map example
Haiyan typhoon track data, from the Japan Meteorological Agency Best Track Data
Digital Typhoon in English, with tracks from many cuurent and past typhoons
Hong Kong observatory Haiyan typhoon track data

La Belle France - Map styling with D3js

When I looked to the vectorial web mapping tools for the first time, I was completely carried away by this example made with Kartograph by the software creator Gregor Aisch.
This example was, actually, the reason I learned Kartograph and published some posts here.
Later, I learned D3js, which was also amazing, but I didn't find a map like La Bella Italia. So I tried to do a similar one myself.

 

The web site

To make the map, I had to reproduce a the result of the Kartograph example to understand how the effects were achieved. Then, get some base data to make the map, and finally, code the example.

Getting the data

The map has, basically, data about the land masses, the countries and the French regions.

I tried data from different places (one of them, Eurostat, that ended in this example), until I decided to use the Natural Earth data. After some attempts, I decided to use the 1:10m data. The files are:

1. ne_10m_admin_0_countries.shp, clipped using:
   ogr2ogr -clipsrc -10 35 15 60 countries.shp ne_10m_admin_0_countries.shp
2. ne_10m_admin_1_states_provinces.shp, clipped using:
   ogr2ogr -clipsrc -10 35 15 60 -where "adm0_a3 = 'FRA'" regions.shp ne_10m_admin_1_states_provinces.shp
3. ne_10m_land.shp, downloaded from github, since the official version gave errors when converted to TopoJSON. Clipped using:
   ogr2ogr -clipsrc -10 35 15 60  land.shp ne_10m_land.shp

With that, the land, countries and regions are available. To merge them into a single TopoJson file, I used:

topojson -o ../data.json countries.shp regions.shp

The html code

Since the code is quite long, and I think that I will made some more posts about specific parts of the technique, the comments are a bit shorter than usually.

CSS

Note how is the font AquilineTwo.ttf  loaded:

@font-face {
    font-family: 'AquilineTwoRegular';
    src: url('AquilineTwo-webfont.eot');
    src: url('AquilineTwo-webfont.eot?#iefix') format('embedded-opentype'),
         url('AquilineTwo-webfont.woff') format('woff'),
         url('AquilineTwo-webfont.ttf') format('truetype'),
         url('AquilineTwo-webfont.svg#AquilineTwoRegular') format('svg');
    font-weight: normal;
    font-style: normal;

}

Later, the font can be set using .attr("font-family","AquilineTwoRegular")

Loading the layers

To achieve the effects, some layers are loaded more than one time, so different filters can be applied to get the shades and blurred borders:

svg.selectAll(".bgback")
    .data(topojson.feature(data, data.objects.land).features)
  .enter()
    .append("path")
      .attr("class", "bgback")
      .attr("d", path)
      .style("filter","url(#oglow)")
      .style("stroke", "#999")
      .style("stroke-width", 0.2);

In this case, the land masses are drawn, applying the effect named oglow, which looks like:

var oglow = defs.append("filter")
  .attr("id","oglow");
  oglow.append("feColorMatrix")
    .attr("in","SourceGraphic")
    .attr("type", "matrix")
    .attr("values", "0 0 0 0 0   0 0 0 0 0   0 0 0 0 0   0 0 0 1 0")
    .attr("result","mask");
  oglow.append("feMorphology")
    .attr("in","mask")
    .attr("radius","1")
    .attr("operator","dilate")
    .attr("result","mask");
  oglow.append("feColorMatrix")
    .attr("in","mask")
    .attr("type", "matrix")
    .attr("values", "0 0 0 0 0.6 0 0 0 0 0.5333333333333333 0 0 0 0 0.5333333333333333  0 0 0 1 0")
    .attr("result","r0");
  oglow.append("feGaussianBlur")
    .attr("in","r0")
    .attr("stdDeviation","4")
    .attr("result","r1");
  oglow.append("feComposite")
    .attr("operator","out")
    .attr("in","r1")
    .attr("in2","mask")
    .attr("result","comp");

To see how svg filters work, many pages are available. I got them looking at the Kartograph example generated html.

Adding the labels

The labels aren't inside the TopoJSON (although they could be!), so I decided the labels to add and put them into an array:

var cities = [ 
                {'pos': [2.351, 48.857], 'name': 'Paris'},
                {'pos':[5.381, 43.293], 'name': 'Marseille'},
                {'pos':[3.878, 43.609], 'name': 'Montpellier'},
                {'pos':[4.856, 45.756], 'name': 'Lyon'}, 
                {'pos':[1.436, 43.602], 'name': 'Toulouse'},
                {'pos':[-0.566, 44.841], 'name': 'Bordeaux'},
                {'pos':[-1.553, 47.212], 'name': 'Nantes'},
                {'pos':[8.737, 41.925], 'name': 'Ajaccio'},
              ];

Then, adding them to the map is easy:

var city_labels =svg.selectAll(".city_label")
    .data(cities)
    .enter();

  city_labels
    .append("text")
    .attr("class", "city_label")
    .text(function(d){return d.name;})
    .attr("font-family", "AquilineTwoRegular")
    .attr("font-size", "18px")
    .attr("fill", "#544")
    .attr("x",function(d){return projection(d.pos)[0];})
    .attr("y",function(d){return projection(d.pos)[1];});


  city_labels
    .append("circle")
    .attr("r", 3)
    .attr("fill", "black")
    .attr("cx",function(d){return projection(d.pos)[0];})
    .attr("cy",function(d){return projection(d.pos)[1];});

Note that the positions must be calculated transforming the longitude and latitude using the d3js projection functions.

The ship

To draw the ship, tow things are necessary, the path and the ship.

To draw the path:

var ferry_path = [[8.745, 41.908],
                  [8.308, 41.453],
                  [5.559, 43.043], 
                  [5.268, 43.187], 
                  [5.306, 43.289]
                  ];
  var shipPathLine = d3.svg.line()
    .interpolate("cardinal")
    .x(function(d) { return projection(d)[0]; })
    .y(function(d) { return projection(d)[1]; });

  var shipPath = svg.append("path")
    .attr("d",shipPathLine(ferry_path))
    .attr("stroke","#000")
    .attr("class","ferry_path");

Basically, d3.svg.line is used to interpolate the points, making the line smoother. This is easier than the Kartograph way with geopaths, where the Bézier control points have to be calculated. d3.svg.line is amazing, more than what I thought before.
I don't know if the way to calculate the projected points is the best one, since I do it twice for each point, which is ugly.
To move the ship, a ship image is appended, and then moved with a setInterval:

  var shipPathEl = shipPath.node();
  var shipPathElLen = shipPathEl.getTotalLength();

  var pt = shipPathEl.getPointAtLength(0);
  var shipIcon = svg.append("image")
          .attr("xlink:href","ship.png")
          .attr("x", pt.x - 10)
          .attr("y", pt.y - 5.5)
          .attr("width", 15)
          .attr("height", 8);

  var i = 0;
  var delta = 0.05;
  var dist_ease = 0.2;
  var delta_ease = 0.9;
  setInterval(function(){
    
    pt = shipPathEl.getPointAtLength(i*shipPathElLen);
    shipIcon
      .transition()
      .ease("linear")
      .duration(1000)
      .attr("x", pt.x - 10)
      .attr("y", pt.y - 5.5);
    
    //i = i + delta;

    if (i < dist_ease){
      i = i + delta * ((1-delta_ease) + i*delta_ease/dist_ease);
    }else if (i > 1 - dist_ease){
      i = i + delta * (1 - ((i - (1 - dist_ease)) * (delta_ease/dist_ease)));
    }else{
      i = i + delta;
    }
    if (i+0.0001 >= 1 || i-0.0001 <= 0)
      delta = -1 * delta;
  },1000);

The ship position is calculated every second, and moved with a d3js transition to make it smooth (calculating everything more often didn't give this smooth effect)
The speed of the ship is changed depending on the proximity to the harbour, to a void the strange effect of the ship crashing into it. The effect is controlled by dist_ease and delta_ease parameters, that change the distance where the speed is changed, and the amount of speed changed.

What's next

• The SVG filters should be explained in  a better way, maybe packing them into functions as Kartograph does.
• SVG rendering lasts quite a lot in my computer. The same happens with Kartograph, so the problem comes from the SVG rendering. Anyway, could be improved.
• A canvas version would be nice.

Links

La Bella Italia -- The example I have used as a model
Gregor Aisch's home page
Natural Earth

Using EUROSTAT's data with D3.js

EUROSTAT is the European Union office for statistics. Its goal is to provide statistics at European level that enable comparisons between regions and countries.

The NUTS classification (Nomenclature of territorial units for statistics) is a hierarchical system for dividing up the economic territory of the EU. Many of the EUROSTAT data is relative to these regions, and this data is a good candidate to be mapped.

The problem is that EUROSTAT provides some shapefiles in one hand, but without the actual names for the regions, nor other information, and some tables  with the name and population for every region. This data has to be joined to be able to map properly.

In this example, he EUROSTAT's criminality statistics are mapped using these provided files.

You can see the working example at bl.ocks.org, and get all the source code at GitHub.
There is also a second example to see how to change the data easily.

Organizing the data

At their site, EUROSTAT provides different Shapefiles for different scales. I think that the best option for maps of all Europe is to use NUTS_2010_10M_SH.zip. Using the 1:60 million scale could be an option, but the file doesn't contain the NUTS3 polygons, so only data at country or big region level is possible.

After downloading the polygons, you will see that no names or population are inside the fields, only the NUTS code. So to make a map, we have to label every polygon. To do it, I downloaded  the NUTS2010.zip file, which contains a csv file with the names for every NUTS region. At the web, it's also possible to find the population in a file, looking for the text Annual average population (1 000) by sex - NUTS 3 regions. Then, I made the following script:

'''
Joins a shp file from EUROSTAT with the xls with the region names so
the resulting shp file has the name of every region and the code.
Roger Veciana, oct 2013
'''
from osgeo import ogr
from os.path import exists
from os.path import basename
from os.path import splitext
from os import remove

def create_shp(in_shp, out_shp, csv_data):
    print "Extracting data"
    
    in_ds = ogr.Open( in_shp )
    if in_ds is None:
        print "Open failed.\n"
        sys.exit( 1 )
    in_lyr = in_ds.GetLayerByName( splitext(basename(in_shp))[0] )
    if in_lyr is None:
        print "Error opening layer"
        sys.exit( 1 )

    if exists(out_shp):
        remove(out_shp)
    driver_name = "ESRI Shapefile"
    drv = ogr.GetDriverByName( driver_name )
    if drv is None:
        print "%s driver not available.\n" % driver_name
        sys.exit( 1 )
    out_ds = drv.CreateDataSource( out_shp )
    if out_ds is None:
        print "Creation of output file failed.\n"
        sys.exit( 1 )
    proj = in_lyr.GetSpatialRef()
    ##Creating the layer with its fields
    out_lyr = out_ds.CreateLayer( 
        splitext(basename(out_shp))[0], proj, ogr.wkbMultiPolygon )
    lyr_def = in_lyr.GetLayerDefn ()
    for i in range(lyr_def.GetFieldCount()):
        out_lyr.CreateField ( lyr_def.GetFieldDefn(i) )

    field_defn = ogr.FieldDefn( "NAME", ogr.OFTString )
    out_lyr.CreateField ( field_defn )

    field_defn = ogr.FieldDefn( "COUNTRY", ogr.OFTString )
    out_lyr.CreateField ( field_defn )

    field_defn = ogr.FieldDefn( "COUNTRY_CO", ogr.OFTString )
    out_lyr.CreateField ( field_defn )

    field_defn = ogr.FieldDefn( "POPULATION", ogr.OFTInteger )
    out_lyr.CreateField ( field_defn )
    

    ##Writing all the features
    in_lyr.ResetReading()
    for feat_in in in_lyr:
        value = feat_in.GetFieldAsString(feat_in.GetFieldIndex('NUTS_ID'))
        if value in csv_data:
            #print csv_data[value]
            feat_out = ogr.Feature( out_lyr.GetLayerDefn())
            feat_out.SetField( 'NUTS_ID', value )
            feat_out.SetField( 'NAME', csv_data[value]['label'] )
            feat_out.SetField( 'POPULATION', csv_data[value]['population'] )
            feat_out.SetField( 'COUNTRY_CO', csv_data[value]['id_country'] )
            feat_out.SetField( 'COUNTRY', csv_data[csv_data[value]['id_country']]['label'] )
            feat_out.SetField( 'STAT_LEVL_', csv_data[value]['level'] )
            feat_out.SetField( 'SHAPE_AREA', feat_in.GetFieldAsString(feat_in.GetFieldIndex('SHAPE_AREA')) )
            feat_out.SetField( 'SHAPE_LEN', feat_in.GetFieldAsString(feat_in.GetFieldIndex('SHAPE_LEN')) )

            feat_out.SetGeometry(feat_in.GetGeometryRef())
            out_lyr.CreateFeature(feat_out)
    in_ds = None
    out_ds = None   

def read_population(csv_file):
    '''
    Reads the NUTS csv population file and returns the data in a dict
    '''
    csv_data = {}
    f = open(csv_file, "r")
    f.readline(); #Skip header
    for line in f:
        line_data = line.split(";")
        try:
            csv_data[line_data[0]] = int(float(line_data[4]) * 1000)
        except Exception, e:
            csv_data[line_data[0]] = -9999
    f.close

    return csv_data
def read_names(csv_file, population_data):
    '''
    Reads a NUTS csv file and returns the data in a dict
    '''
    csv_data = {}
    f = open(csv_file, "r")
    f.readline(); #Skip header
    for line in f:
        line_data = line.split(";")
        csv_data[line_data[0]] = {'label': line_data[1],
            'level': line_data[2],
            'id_country': line_data[3],
            'code_country': line_data[4],
            'population': 0}
        if line_data[0] in population_data:
            csv_data[line_data[0]]['population'] = population_data[line_data[0]]

    f.close

    return csv_data

if __name__ == '__main__':
    population_data = read_population('demo_r_d3avg_1_Data.csv')
    csv_data = read_names('NUTS_2010.csv', population_data)
    create_shp('NUTS_2010_10M_SH/Data/NUTS_RG_10M_2010.shp', 'regions.shp', csv_data)

• Basically, we iterate for every polygon in the shp file, and look into the csv data for the attributes to add. 
• Population data and region name are in tow separate files. To read them, there are two functions: read_population and read_names.
• create_shp function reads the geometries, joins them with the population and name, and finally writes the output file.
• Notice that NUTS regions overlap. There are levels 1, 2 and 3. The first one is at country level, the second takes big areas from the countries, and the third one are the actual departments in every country. To map any data is necessary to choose the level at which the map has to be represented, or mix the different levels if it's necessary, as in this case.

To create a map, I chose the criminality data. The file is again at EUROSTAT, under the text Crimes recorded by the police - NUTS 3 regions. Despite its name, some regions are NUTS3, and some NUTS2. Most of  the files from the site labeled with NUTS regions should work with only little changes in the map code.

Creating a D3.js map to show the data

To draw the data in a map I've used a similar solution as the one in the post D3js Electoral Map:

Crimes recorded by the police - NUTS 3 regions









Source: eurostat

• Lines 5 to 18 give style to the map background and the tooltip
• Lies 29 to 43 set up the map, with its size, projection and svg element
• Line 45 sets the color scale (this is different to the electoral map example). The colors will vary from blue to red. Discrete colors are an other option. Just changing this line, you can change the whole map colors.
• Lines 47 to 53 read and parse the csv with the criminality data into  an object (we will only use 2010 data)
• Line 55 downloads the TopoJSON containing the NUTS regions
• Then lines 56 to 78, the map is actually drawn. Note
• Line 60 filters only the regions that have data in the CSV file. For some countries, this is the NUTS2 region (see Italy) and for others, NUTS3 (like Germany). The following lines are only run if there is data (see D3js filter)
• Line 66 sets the color of the region. Note that calls the scale defined at line 45, and calculates the number by multiplying the crimes by 100000 and dividing by the region population to get the density
• Line 71 sets the region opacity to 0 if there is no data for it. With the filter could be avoided
• Lines 78 and 81 set the tooltip box, taken from this example.
• Line 98 calls the function that draws the legend. The function is in a separate file to make it easy to reuse

 

var legend = function(accessor){
    return function(selection){
      
      //Draw the legend for the map
      var legend = selection.append("g");
      var legendText = accessor.title;
      var numSquares = accessor.elements;
      var xLegend = 0;
      var yLegend = 0;
      var widthLegend = 400;
      
      var title_g = legend.append("g");
      var values_g = legend.append("g");
      
      var legendTitle = title_g.append("text")
        .text("Legend")
        .attr("font-family", "sans-serif")
        .attr("font-size", "18px")
        .attr("fill", "black");
        var bbox = legendTitle.node().getBBox();
        legendTitle.attr("x", xLegend + widthLegend/2 - bbox.width/2);
        legendTitle.attr("y", yLegend + 20);
        
      var legendTextEl = title_g.append("text")
        .text(legendText)
        .attr("y", yLegend + 75)
        .attr("font-family", "sans-serif")
        .attr("font-size", "14px")
        .attr("fill", "black");
      var bbox = legendTextEl.node().getBBox();
      legendTextEl.attr("x", xLegend + widthLegend/2 - bbox.width/2);

      for (i=0; i < numSquares; i++){
        values_g.append("rect")
          .attr("x", xLegend + (i+0.1*i/numSquares)*(widthLegend/numSquares))
          .attr("y", yLegend + 25)
          .attr("width", (widthLegend/numSquares)*0.9)
          .attr("height", 20)
          .attr("fill", color(accessor.domain[0] + i * accessor.domain[1]/(numSquares-1)));
        var value_text = values_g.append("text")
          .text(accessor.domain[0] + i * accessor.domain[1]/(numSquares-1))
          .attr("y", yLegend + 55)
          .attr("font-family", "sans-serif")
          .attr("font-size", "12px")
          .attr("fill", "black");
        var bbox = value_text.node().getBBox();
        value_text
          .attr("x", xLegend + (i+0.1*i/numSquares)*(widthLegend/numSquares) + (widthLegend/numSquares)*(0.9/2)- bbox.width/2);

      }
      var scale = accessor.width / 400;
      legend.attr("transform","translate("+accessor.posx+","+accessor.posy+") scale("+scale+") ");

};
};

•  Note how the function is called. The argument accessor is the one thas have the object properties (if called from a geometry, for instance), and selection is the node where the function is called from.
• The code is not very D3jsnic, using things as a for loop but it works and can be reused for drawing other legends from a D3js scale.

  Related posts and sites

• D3js Electoral Map
• D3js filters
• D3js world map
• Tooltip as a D3 helper
• EUROSTAT
• Crime rate in Europe by EUROSTAT NUTS regions
• Touristic establishments by EUROSTAT NUTS regions

Castor project's earthquakes map with D3js

The Castor project is a submarine natural gas storage facility located in front of the eastern Iberian Peninsula coast. The idea is to store the Algerian gas inside an old oilfield cavity. At least this is what I understood (sorry, geologists).
Somehow, when the facility started working, a series of earthquakes have started to occur. At the beginning, the platform owners said it wasn't related to their activity, but now everybody agrees that it is, and the activity has stopped, but not the earthquakes.

I didn't find a nice map about the earthquakes epicenters, so I thought that D3js would be a good option to do it.

The star represents the aproximate platform location, and the circles, the epicenters. It's easy to see why they are related to the platform activity.

The animated map is at my bl.ocks.org page. The explanations are in Catalan, but basically say the same as here.

Getting the data

The data about the significant earthquakes around Catalonia can be found at the Catalan Geologic Institute web site, but the format for the reports is not very convenient to get the data, so I made this python script to get it:

# -*- coding: utf-8 -*-
#http://jramosgarcia.wordpress.com/2013/10/01/que-es-el-proyecto-castor/
import urllib
import json
import dateutil.parser
import datetime

def get_data(url):
    filehandle = urllib.urlopen(url)
    html = filehandle.read()
    filehandle.close()

    lines = html.splitlines()

    for i in range(len(lines)):
    
        if lines[i].find('Latitud') > 0:
            lat = lines[i].strip().split(" ")[1].replace("º","")
        if lines[i].find('Longitud') > 0:
            lon = lines[i].strip().split(" ")[1].replace("º","")   
        if lines[i].find('mol del dia') > 0:
            date = lines[i + 1].strip().replace(" >/div<","")
        if lines[i].find('Hora origen') > 0:
            hour = lines[i].strip().split(" ")[4]                   
        if lines[i].find('Magnitud') > 0:
            magnitude = lines[i+1].strip().split(" ")[0]

    date_array  = date.split("/")
    hour_array = hour.split(":")

    date_time = datetime.datetime(int(date_array[2]),int(date_array[1]),int(date_array[0]),int(hour_array[0]), int(hour_array[1]))

    data = {'lat':lat, 'lon':lon, 'datetime': date_time.isoformat(), 'magnitude': magnitude}
    return json.dumps(data)

if __name__ == "__main__":
    url_list = [
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130910175510/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130913095842/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130918142943/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130920104607/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130924091301/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130925125029/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130929084140/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130929192416/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130930005900/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20130930051316/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131001045206/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131001055709/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131002121626/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131002232928/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131003012732/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131003031301/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131004113222/comact.html',
    'http://www.igc.cat/web/gcontent/ca/sismologia/sismescomact/comhistcat/20131004120323/comact.html'
    ]

    f = open("data.json","w")
    f.write("[")
    json_data = ""
    for url in url_list:
     json_data = json_data + get_data( url ) + ", "
       
    f.write(json_data[0:-2])
    f.write("]")
    f.close()
   

• The web pages are in the list, since the interesting reports have to be chosen one by one. It would be nice to have a better way to do it.
• Then, the get_data function just parses the text in the way that all the reports are parsed properly.The data is stored in a json file to make easier it's use from D3js.

Using D3js to visualize the data

I used this example by Mike Bostock to create the background  map. The tiles origin has been changed because the example tiles are not available at this zoom level, and to have more points of interest to situate the earthquake locations.

This is the code:

<script>
/**
Based on Mike Bostock's http://bl.ocks.org/mbostock/4150951
*/
var width = 960,
    height = 500;

var projection = d3.geo.mercator()
    .center([0.5491, 40.4942])
    .scale(20000);

var path = d3.geo.path()
    .projection(projection);

var tile = d3.geo.tile()
    .scale(projection.scale() * 2 * Math.PI)
    .translate(projection([0, 0]))
    .zoomDelta((window.devicePixelRatio || 1) - .5);

var svg = d3.select("body").append("svg")
    .attr("width", width)
    .attr("height", height);

  var tiles = tile();

  var defs = svg.append("defs");

var magnitudeScale = d3.scale.linear().domain([2,5]).range([5,30]);
d3.json("data.json", function(error, locations) {

  svg.append("g")
      
    .selectAll("image")
      .data(tiles)
    .enter().append("image")
      .attr("xlink:href", function(d) { return "http://" + ["a", "b", "c", "d"][Math.random() * 4 | 0] + ".tiles.mapbox.com/v3/examples.map-vyofok3q/" + d[2] + "/" + d[0] + "/" + d[1] + ".png"; })
      .attr("width", Math.round(tiles.scale) + 1)
      .attr("height", Math.round(tiles.scale) + 1)
      .attr("x", function(d) { return Math.round((d[0] + tiles.translate[0]) * tiles.scale); })
      .attr("y", function(d) { return Math.round((d[1] + tiles.translate[1]) * tiles.scale); });

  svg.append("g")
      .append('path')
      .attr("d","m 0,0 -8.47858,-5.22254 -8.31623,5.47756 2.34696,-9.67752 -7.77927,-6.21653 9.92909,-0.75852 3.50829,-9.31953 3.78972,9.20873 9.94748,0.45679 -7.58687,6.44982 z")
      .attr("stroke","black")
      .attr("stroke-width",2)
      .style("fill", d3.rgb(90, 90, 90))
      .attr("transform", "translate("+projection([0.66879, 40.33503])[0]+","+projection([0.66879, 40.33503])[1]+")");
  
  var locationsG = svg.append("g");
  

  function addLocation(loc){
    locationsG.append('circle')
      .attr('class','location')
      .attr("r", 5)
      .attr("cx", projection([loc.lon, loc.lat])[0])
      .attr("cy", projection([loc.lon, loc.lat])[1])
      .style("fill", d3.rgb(255, 0, 0).darker(2))
      .style("opacity", 0.8)
      .transition()
      .duration(1000)
      .attr("r", magnitudeScale(loc.magnitude))
      .transition()
      .delay(2000)
      .duration(2000)
      .style("opacity",0.3);

    locationsG
      .append("text")
      .text(loc.magnitude)
      .attr("x", projection([loc.lon, loc.lat])[0] - 10)
      .attr("y", projection([loc.lon, loc.lat])[1] + 5)
      .attr("font-family", "sans-serif")
      .attr("font-size", "12px")
      .attr("fill", "black")
      .style("opacity",0)
      .transition()
      .duration(1000)
      .style("opacity",1)
      .transition()
      .delay(2000)
      .duration(2000)
      .style("opacity",0);
  }

  //addLocation({"lat": "40.43", "magnitude": "2.7", "lon": "0.7", "datetime": "2013-10-09T06:43:16"});
  
  var intDate = new Date("2013-09-10T00:00:00Z");
  var maxDate = new Date("2013-10-04T00:00:00Z");
  var usedLocations = new Array();

  var dateTitle = svg
    .append("text")
    .attr("id", "dataTitle")
    .text(intDate.toLocaleDateString())
    .attr("x", 70)
    .attr("y", 20)
    .attr("font-family", "sans-serif")
    .attr("font-size", "20px")
    .attr("fill", "black");

  var interval = setInterval(function() {

    dateTitle.text(intDate.toLocaleDateString());

    for (i = 0; imaxDate){
        clearInterval(interval);
      }

    }
    
    intDate.setDate(intDate.getDate() + 1);
  }, 1000);
  
});

•  Lines 30 and 48 are the most important ones to create the tile map. In this case, no zoom or pan is set, so the code is quite simple.
• The line 57 creates the star indicating the platform location. I made the star using Inkscape and captured the code using the tool to see the generated svg. To situate the symbol on the map, a transform attribute is used, translating the symbol to the pixel calculated projecting the location coordinates.
• To add the earthquake epicenters:
• The function addLocation (line 68) adds the circle and the text indicating the earthquake magnitude. To locate them, the projection function is used again. Two transitions are used to make the circle grow and then get brighter. The same for making the text disappear.
• An interval is created (line 118) to change the date (one day per second). For every iteration, the date is evaluated and the earthquakes occured during this day plotted using the addLocation function. I didn't find a "more D3js" solution. The earthquakes already plotted are stored in an array to avoid plotting them more than once.
  The date on the map is also changed for every iteration.

Some related posts and pages:

• D3.geo.tile
• D3 World maps 
• Let's make a map
• Three little circles
• Que es el proyecto Castor? (in Spanish). Very good explanation about the Castor project, with some illustrations for the non-Spanish speakers.
• Quakes rattle Spain, offshore gas storage project blamed

Flag map with D3js

 In this example I'll show how to draw a map filling the polygons with images. In this case, Western Africa contries filled with their flags:

The zone represented is this one:

As usual, you can get all the code at GitHub:

• Canvas version
• SVG version

Canvas

The code basically uses Mike Bostock's queue.js to download all the flag images from the wikipedia, the world borders and the contries names. Once everyting is donwloaded, a function is triggered, that draws the flags, clipping them by the country shape. The first version is written using Canvas instead of SVG, just to see how the clipping and loading of external images is done.

The code:

• I make an array with all the image locations, indexing by name to make it easier to understand, even though the map file doesn't contain names but indexes.
• Lines 37 to 47 configure the map projection and the Canvas element.
• Lines 54 to 77 load all the elements. Since loading an image is not a function in D3js, a custom function has to be created, returning the error or the success value. If images are not preloaded, the code will try to paste them anyway, so the country flag won't be drawn
• Function ready is triggered when everything is loaded, and it actually draws the map.
• The error is handled at the first lines (very poorly, I know, a nodata image could be loaded, or something like that)
• Until line 97, the base map is loaded (the land gets hidden later by the flags).
• Lines 100 to 109 get the countries from the list, looks for them in the countries file by using a filter function and creates the images we are going to use later.
• At line 111, all the flags are pasted on the map.
• context.save();  Saves what has been drawn until the moment, so new things can happen
• Then, the country border is drawn in a path, without using any color or fill, because then, context.clip(); is called to use this path as the clip path.
• The image is pasted, using the drawImage method. The bounds are recalculated before for the countries that appera cut at the map.
• context.restore(); frees the clipping.
• Finally, at line 130, the borders are drawn to make the map prettier.

SVG

The SVG version is simplier, because no image preloading is needed, since the svg contains only the image source, and the browser will draw it when the image loads.

• The code is almost the same, just that it doesn't preload the images with queue.js
• Obviously, the canvas tag is replaced by SVG, and to draw, the usual SVG mode is used.
• To clip, a clipPath is defined with the country shape and an id, so it can be assigned to an image. 
• The image is added using an image tag
• xlink:href sets the image link
• clip-path sets the path used to clip the image
• preserveAspectRatio has to be set as none, so the image ratio can be changed to make the flag fit the bounding box.

Notes:

To preload images using queue.js I asked this question at StackOverflow, kindly
answered by explunit.