## Saturday, October 11, 2014

### Basemap raster clipping with a shapefile

Basemap is a great library for mapping faster than other python options, but there are some usual things I couldn't find how to do. Clipping a raster using a shape is one of them. Here's how do I do it
 The output
As usual, all the code can be found at GitHub

### Getting some data

The example plots some elevation data, taken from the SRTM. After looking for some options, the easiest to work with was this one: http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp
The shapefile will be the border of Andorra, taken from Natural Earth
The result is a little poor because the resolution is low, but works well for the example.

### The script

from mpl_toolkits.basemap import Basemap
from matplotlib.path import Path
from matplotlib.patches import PathPatch
import matplotlib.pyplot as plt
from osgeo import gdal
import numpy
import shapefile

fig = plt.figure()

for shape_rec in sf.shapeRecords():
if shape_rec.record[3] == 'Andorra':
vertices = []
codes = []
pts = shape_rec.shape.points
prt = list(shape_rec.shape.parts) + [len(pts)]
for i in range(len(prt) - 1):
for j in range(prt[i], prt[i+1]):
vertices.append((pts[j][0], pts[j][1]))
codes += [Path.MOVETO]
codes += [Path.LINETO] * (prt[i+1] - prt[i] -2)
codes += [Path.CLOSEPOLY]
clip = Path(vertices, codes)
clip = PathPatch(clip, transform=ax.transData)

m = Basemap(llcrnrlon=1.4,
llcrnrlat=42.4,
urcrnrlon=1.77,
urcrnrlat=42.7,
resolution = None,
projection = 'cyl')

ds = gdal.Open('srtm_37_04.tif')

gt = ds.GetGeoTransform()
x = numpy.linspace(gt[0], gt[0] + gt[1] * data.shape[1], data.shape[1])
y = numpy.linspace(gt[3], gt[3] + gt[5] * data.shape[0], data.shape[0])

xx, yy = numpy.meshgrid(x, y)

cs = m.contourf(xx,yy,data,range(0, 3600, 200))

for contour in cs.collections:
contour.set_clip_path(clip)

plt.show()
• I used the pyshp library for reading the shapefile, since Fiona and GDAL don't work well together, and OGR was longer
• Lines 14 to 27 create the path. A Matplotlib path is made by two arrays. One with the points (called vertices in the script), and the other with the functions for every point (called codes)
• In our case, only straight lines have to be used, so there will be a MOVETO to indicate the beginning of the polygon, many LINETO to create the segments and one CLOSEPOLY for closing it
• Of course, only the polygon for Andorra has to be used. I get it from the shapefile attributes
•  The prt array is for managing multipolygons, which is not the case, but the code will create correct clipping for multipolygons
• The path is created using the Path function, and then added to a PathPatch, to be able to use it as a closed polygon. Note the trasnform=ax.transData attribute. This assumes the polygon coordinates to be the ones used in the data (longitudes and latitudes in our case). More information here
• Next code lines draw the map as usual. I have used a latlon projection, so all the values for the raster and shapefile can be used directly. If the output raster was in an other projection, the shapefile coordinates should be appended to the path using the output projection (m(pts[j][0], pts[j][1]))
• The x and y coordinates are calculated from the GDAL geotransform, and then turned into a matrix using meshgrid
• The clipping itself is made in the lines 48 and 49. For each drawn element, the method set_clip_path is applied

## Saturday, August 16, 2014

### Shortest distance to a geometry in a specified direction using Python

Looking at this map, I wondered how to calculate which geometry in a set is the closest to a point in a given direction.
Usually, the problem is finding the closest geometry in general, which is easy using the distance function, but I couldn't find a solution for this other.

So I put me this problem: Which is the closest country that I have at each direction, knowing my geographical coordinates?
All the source code is, as usual, at GitHub

### The algorithm

The main idea is:
1. Create an infinite line from the point towards the desired direction.
2. Calculate the difference geometry between the  line and each polygon
1. If the polygon and the line actually intersect, the result will be a multi-line. The first line length of the multi-line is the distance we are looking for
So this would be the initial situation:
And the distance to the polygon 1 would be calculated as:
The main problem is how to calculate the difference between the two geometries, but fortunately, shapely comes with this function, so coding it is not so difficult:
from shapely.geometry import Polygon
from shapely.geometry import LineString
from math import cos
from math import sin
from math import pi

def closest_polygon(x, y, angle, polygons, dist = 10000):

angle = angle * pi / 180.0
line = LineString([(x, y), (x + dist * sin(angle), y + dist * cos(angle))])

dist_min = None
closest_polygon = None
for i in range(len(polygons)):
difference = line.difference(polygons[i])
if difference.geom_type == 'MultiLineString':
dist = list(difference.geoms)[0].length
if dist_min is None or dist_min > dist:
dist_min = dist
closest_polygon = i

return {'closest_polygon': closest_polygon, 'distance': dist_min}

if __name__ == '__main__':

polygons = []
polygons.append(Polygon([(4, 2), (4, 4), (6, 4), (6, 2)]))
polygons.append(Polygon([(7, 2), (7, 4), (9, 4), (9, 2)]))

print closest_polygon(3, 3, 90, polygons)
• The main section creates the two squares using shapely
• The closest_polygon function calculates the closest polygon and its distance:
• A LineString to the desired direction is calculated. The dist is in the units used by the polygons. An infinite line isn't possible, so the distance must be larger than the further
• For each of the polygons to analyze, the difference is calculated using the shapely difference method
• Then, if the line and the polygon intersect (and the line is long enough), a MultilineString will be the result of the difference operation. The first String in the MultilineString is the one that connects our point with the polygon. Its length is the distance we are looking for
 The example schema, drawn with the script draw_closest.py

### Calculating the closest country in each direction

After getting the formula for calculating the closest polygon, the next step would be using it for something. So:
Which country do I have in all directions?
To create the script, some things have to be considered:
1. The projection should be azimuthal equidistant so the distances can be compared in all the directions from the given point
2. I've used the BaseMap library to draw the maps. I find it a bit tricky to use, but the code will be shorter
The script is used this way:

usage: closest_country.py [-h] [-n num_angles] [-o out_file] [-wf zoom_factor]
lon lat

Creates a map with the closest country in each direction

positional arguments:
lon              The point longitude
lat              The point latitude

optional arguments:
-h, --help       show this help message and exit
-n num_angles    Number of angles
-o out_file      Out file. If present, saves the file instead of showing it
-wf zoom_factor  The width factor. Use it to zoom in and out. Use > 1 to
draw a bigger area, and <1 for a smaller one. By default is
1

For example:
python closest_country.py -n 100 -wf 2.0 5 41
The code has some functions, but the main one is draw_map:
def draw_map(self, num_angles = 360, width_factor = 1.0):

#Create the map, with no countries
self.map = Basemap(projection='aeqd',
lat_0=self.center_lat,lon_0=self.center_lon,resolution =None)
#Iterate over all the angles:
results = {}
distances = []
for num in range(num_angles):
angle = num * 360./num_angles
closest, dist = self.closest_polygon(angle)
if closest is not None:
distances.append(dist)
if (self.names[closest] in results) == False:
results[self.names[closest]] = []

results[self.names[closest]].append(angle)

#The map zoom is calculated here,
#taking the 90% of the distances to be drawn by default
width = width_factor * sorted(distances)[
int(-1 * round(len(distances)/10.))]

#Create the figure so a legend can be added
plt.close()
fig = plt.figure()
cmap = plt.get_cmap('Paired')

self.map = Basemap(projection='aeqd', width=width, height=width,
lat_0=self.center_lat,lon_0=self.center_lon,resolution =None)

#Fill background.
self.map.drawmapboundary(fill_color='aqua')

#Draw parallels and meridians to give some references
self.map.drawparallels(range(-80, 100, 20))
self.map.drawmeridians(range(-180, 200, 20))

#Draw a black dot at the center.
xpt, ypt = self.map(self.center_lon, self.center_lat)
self.map.plot([xpt],[ypt],'ko')

#Draw the sectors
for i in range(len(results.keys())):
for angle in results[results.keys()[i]]:
anglerad = float(angle) * pi / 180.0
anglerad2 = float(angle + 360./num_angles) * pi / 180.0
polygon = Polygon([(xpt, ypt), (xpt + width * sin(anglerad), ypt + width * cos(anglerad)), (xpt + width * sin(anglerad2), ypt + width * cos(anglerad2))])
patch2b = PolygonPatch(polygon, fc=cmap(float(i)/(len(results) - 1)), ec=cmap(float(i)/(len(results) - 1)), alpha=1., zorder=1)

#Draw the countries
for polygon in self.polygons:
patch2b = PolygonPatch(polygon, fc='#555555', ec='#787878', alpha=1., zorder=2)

#Draw the legend
cmap = self.cmap_discretize(cmap, len(results.keys()))
mappable = cm.ScalarMappable(cmap=cmap)
mappable.set_array([])
mappable.set_clim(0, len(results))
colorbar = plt.colorbar(mappable, ticks= [x + 0.5 for x in range(len(results.keys()))])
colorbar.ax.set_yticklabels(results.keys())

plt.title('Closest country')

• The first steps are used to calculate  the closest country in each direction, storing the result in a dict. The distance is calculated using the closest_polygon method, explained in the previous section..
• The actual map size is then calculated, taking the distance where the 90% of the polygons will appear. The width_factor can change this, because some times the result is not pretty enough. Some times has to much zoom and some, too few. Note that the aeqd i.e., Azimuthal Equidistant projection is used, since is the one that makes the distances in all directions comparable.
• Next steps are to actually drawing the map
• The sectors (the colors indicating the closest country) are drawn using the Descartes library and it's PolygonPatch
• The legend needs to change the color map to a discrete color map. I used a function called cmap_discretize, found here, to do it
• The legend is created using the examples found in this cookbook
Some outputs:

### Next steps: What's across the ocean

Well, my original idea was creating a map like this one, showing the closest country when you are at the beach. Given a point and a direction (east or west in the example), calculating the country is easy, and doing it for each point in the coast is easy too. The problem is that doing it automatic is far more difficult, since you have to know the best direction (not easy in many places like islands), which countries to take as the origin, etc.
An other good thing would be doing the same, but with d3js, since  the point position could become interactive. I found some libraries like shapely.js or  jsts, but I think that they still don't implement the difference operation that we need.

A LinkedIn discussion that gave me some ideas

How to install Basemap (you can use a virtual environment to test it without installing it in the whole system). Be sure to have pip installed, and the python-dev package in case you are using Ubuntu. Some distributions have Basemap as a system package too.
How to create an Azimuthal equidistant map with Basemap - The Azimuthal Equidistant projection
Some simple and useful Basemap examples
Advanced Basemap tricks that helped me to add the legend and much more
How to discretize a color map
Descartes: Drawing polygons in Matplotlib

## Monday, July 7, 2014

### Using the D3 trail layout to draw the Hayian tracks

I wrote many examples (1, 2, 3 and 4) and some entries in the blog (1 and 2) showing how to draw animated paths on a map using the D3 library.
But since then, Benjamin Schmidt wrote a D3 layout, called trail layout, that simplifies a lot doing this kind of stuff.
Since the layout is new, and hasn't got many examples (actually, two made by the author), I'll try to show how to work with it.

### The trail layout

How does the trail layout work? The author defines it as:
This is a layout function for creating paths in D3 where (unlike the native d3.svg.line() element) you need to apply specific aesthetics to each element of the line.
Basically, the input is a set of points, and the layout takes them and creates separate segments to join them. This segments can be either line or d SVG elements.

#### Let's see the simplest example:



var width = 600,
height = 500;

var points = [{"x":0,"y":0}, {"x":200,"y":200}, {"x":0,"y":400}, {"x":200,"y":100}];
var svg = d3.select("body").append("svg")
.attr("width", width)
.attr("height", height);

var trail = d3.layout.trail().coordType('xy');

var trail_layout = trail.data(points).layout();

paths = svg.selectAll("line").data(trail_layout);

paths.enter()
.append('line')
.style("stroke-width",3)
.style("stroke","black")
.attr("x1",function(d) {return d.x1})
.attr("y1",function(d) {return d.y1})
.attr("y2",function(d) {return d.y2})
.attr("x2",function(d) {return d.x2})

• In this case, the points are defined as an array of objects with the x and y properties. If the x and y are named this way, the layout takes them directly. If they are called for instance lon and lat, the layout must be told how to get them.
• Line 10 creates the SVG
• Line 14 initializes the layout. In this case, the layout is using the coordType xy, which means that as a result will give the initial and end point for each segment, convenient for drawing SVG line elements. The other option is using the coordinates value, which is convenient for drawing d elements, as we will see later.
•  Line 15 is where the data is set and the layout is retrieved
• The last step is where the lines are actually drawn.
• For each data element, the line svg is added
• The styles are applied
• The extremes of the line are set using the attributes x1, y1, x2, y2

#### How to use coordinates as the coordType:

The following example created the trail as a set of SVG line elements, but the trail layout has an option for creating it as a set of SVG d elements (paths).
You can see the example here. The data, in this case, is the Hayian track. As you can see, it's quite similar as the former example, with the following differences:
• Since in this case we are using geographical coordinates, a projection must be set, and also a d3.geo.path to convert the data into x and y positions, as usual when drawing d3 maps
• When initializing the trail layout, coordinates must be set as the coordType.
• Since the data elements do not store the positions with the name x and y, the layout has to be told how the retrieve them using the positioner:
.positioner(function(d) {return [d.lon, d.lat];})
• When drawing the trail, a path element is appended instead the line element, and the d attribute is set with the path  function defined above.

### Creating the map with the trail

Once the basic usage of the trail layout is known, let's reproduce the Hayian path example (simplified for better understanding):


.map {
fill: none;
stroke: #777;
stroke-opacity: .5;
stroke-width: .5px;
}

.land {
fill: #999;
}

.boundary {
fill: none;
stroke: #fff;
stroke-width: .5px;
}

var width = 600,
height = 500;

var projection = d3.geo.mercator()
.scale(5*(width + 1) / 2 / Math.PI)
.translate([width / 2, height / 2])
.rotate([-125, -15, 0])
.precision(.1);

var path = d3.geo.path()
.projection(projection);

d3.json("/mbostock/raw/4090846/world-50m.json", function(error, world) {
d3.json("track.json", function(error, track) {

var color_scale = d3.scale.quantile().domain([1, 5]).range(colorbrewer.YlOrRd[5]);

var svg = d3.select("body").append("svg")
.attr("width", width)
.attr("height", height);

var trail = d3.layout.trail()
.positioner(function(d) {return projection([d.lon,d.lat]);})
.coordType('xy');

var trail_layout = trail.data(track).layout();

svg.insert("path", ".map")
.datum(topojson.feature(world, world.objects.land))
.attr("class", "land")
.attr("d", path);

svg.insert("path", ".map")
.datum(topojson.mesh(world, world.objects.countries, function(a, b) { return a !== b; }))
.attr("class", "boundary")
.attr("d", path);

var hayan_trail = svg.selectAll("d").data(trail_layout);

hayan_trail.enter()
.append('line')
.attr("x1",function(d) {return d.x1})
.attr("x2",function(d) {return d.x1})
.attr("y1",function(d) {return d.y1})
.attr("y2",function(d) {return d.y1})
.attr("class","line")
.style("stroke-width",4)
.attr("stroke", function(d){return color_scale(d.class);})
.transition()
.ease("linear")
.delay(function(d,i) {return i*500})
.duration(500)
.attr("x2",function(d) {return d.x2})
.attr("y2",function(d) {return d.y2})
;

});
});

• The map creation is as usual (explained here)
• Lines 49 to 51 create the trail layout as in the former example
• Line 67 creates the trail, but with some differences:
• the beginning and the end of the line are the same point at the beginning, so the line is not drawn at this moment (lines 69 to 72)
• The stroke colour is defined as a function of the typhoon class using the colour scale (line 75)
• A transition is defined to create the effect of the line drawing slowly
• The ease is defined as linear, important in this case where we join a transition for each segment.
• The delay is set to draw one segment after the other. The time (500 ms) must be the same as the one set at duration
• Finally, the changed values are x2 and y2, that is, the final point of the line, which are changed to their actual values
• The complete example, with the typhoon icon and the date is also available
It's possible to use paths instead of lines to draw the map, as in the first version. The whole code is here, but the main changes are in the last section:
hayan_trail.enter()
.append('path')
.attr("d", path)
.style("stroke-width",7)
.attr("stroke", function(d){return color_scale(d.class);})
.style('stroke-dasharray', function(d) {
var node = d3.select(this).node();
if (node.hasAttribute("d")){
var l = d3.select(this).node().getTotalLength();
return l + 'px, ' + l + 'px';
}
})
.style('stroke-dashoffset', function(d) {
var node = d3.select(this).node();
if (node.hasAttribute("d"))
return d3.select(this).node().getTotalLength() + 'px';
})
.transition()
.delay(function(d,i) {return i*1000})
.duration(1000)
.ease("linear")
.style('stroke-dashoffset', function(d) {
return '0px';
});
• The strategy here is to change the stroke-dasharray  and stroke-dashoffset style values as in this example, and changing it later so the effect takes place.
• At the beginning, both values are the same length as the path. This way, the path doesn't appear. The length is calculated using the JavaScript function getTotalLength
• After the transition, the stroke-offset value will be 0, and the path is fully drawn

### Conclusion

I recommend using the trail layout instead of the method from my old posts. It's much cleaner, fast, easy, and let's changing each segment separately.
The only problem I find is that when the stroke width gets thicker, the angles of every segment make strange effects, because the union between them doesn't exist.

This didn't happen with the old method. I can't imagine how to avoid this using lines, but using the coordinates option could be solved transforming the straight lines for curved lines.

## Wednesday, April 16, 2014

### D3 map Styling tutorial IV: Drawing gradient paths

After creating the last D3js example, I was unsatisfied with the color of the path. It changed with the typhoon class at every moment, but it wasn't possible to see the class at every position. When I saw this example by Mike Bostock, I found the solution.

### Understanding the gradient along a stroke example

First, how to adapt the Mike Bostock's Gradient Along Stroke example to a map.
The map is drawn using the example Simple path on a map, from this post. The only change is that the dashed path is changed with the gradient.
You can see the result here.
The differences from drawing a simple path start at the line 100:
var line = d3.svg.line()
.interpolate("cardinal")
.x(function(d) { return projection([d.lon, d.lat])[0]; })
.y(function(d) { return projection([d.lon, d.lat])[1]; });

svg.selectAll("path")
.enter().append("path")
.style("fill", function(d) { return color(d.t); })
.style("stroke", function(d) { return color(d.t); })
.attr("d", function(d) { return lineJoin(d[0], d[1], d[2], d[3], trackWidth); });

•  The line definition remains the same. From every element it gets, it takes the lat and lon attributes, projecting them, and assigning them to the x and y path properties
• A color function is defined at line 41, which will interpolate the color value from green to red:
var color = d3.interpolateLab("#008000", "#c83a22");
• The data is not the line(track) directly, as in the former example, but passed through the functinos sample and quad.
• The sample function assigns a property t with values between 0 and 1, which is used to get the color at every point.
• Finally, the function lineJoin is used to draw a polygon for the sampled area.
The functions used in the Mike Bostock's example aren't explained, I'll try to do it a little:
• sample takes a line (the data applied to a line function), and iterates with the precision parameter as increment along the string, creating an array with all the calculated points.
• quad takes the points calculated by the sample function and returns an array with the adjacent points (i-1, i, i+1, i+2).
• lineJoin takes the four points generated by quad, and draws the polygon, with the help of lineItersect and perp functions.

### Drawing the typhoon track with the colors according to the typhoon class

The final example draws the typhoon path changing smoothly the color according to the typhoon class.
The animation of the path, and the rotating icon are explained in the third part of the tutorial. In this case, the way to animate the path will change.
For each position of the typhoon, a gradient path is drawn, because the gradient is always between two colors. So the part of the code that changes is:
      //Draw the path, only when i > 0 in otder to have two points
if (i>0){
color0 = color_scale(track[i-1].class);
color1 = color_scale(track[i].class);

var activatedTrack = new Array();

activatedTrack.push(track[i-1]);
activatedTrack.push(track[i]);

var color = d3.interpolateLab(color0, color1);
path_g.selectAll("path"+i)
.enter().append("path")
.style("fill", function(d) { return color(d.t);})
.style("stroke", function(d) { return color(d.t); })
.attr("d", function(d) { return lineJoin(d[0], d[1], d[2], d[3], trackWidth); });
}

i = i + 1;
if (i==track.length)
clearInterval(animation)

• Inside the animation interval (line 145), the gradient path is create for each position (starting with the second one to have two points)
• The two colors are taken from the point information
• An array with the two points is created, with the name activatedTrack. I tried using more points, but the result is very similar.
• The color interpolation is calculated (line 172)
• The gradient colored path is created (line 173). Note that the name is path+i, to make different paths each iteration, and not to overwrite them. The method is the same as the one used in the first section.
Besides, an invisible path with all the positions is created, so the typhoon icon can be moved as it was in the third part of the tutorial.

## Monday, March 31, 2014

### Slides for the workshop "Introduction to Python for geospatial uses"

Last 26th, 27th and 28th of March, the 8as Jornadas SIG Libre were held in Girona, where I had the opportunity to give a workshop about Python for geospatial uses.

The slides in Spanish:
http://rveciana.github.io/introduccion-python-geoespacial

The Slides in English:
http://rveciana.github.io/introduccion-python-geoespacial/index_en.html

The example files in both languages:
https://github.com/rveciana/introduccion-python-geoespacial

The meeting was awesome, if you have the opportunity and understand Spanish, come next year!