jueves, abril 28, 2011

Organizing your algorithms

Benjamin Ducke posted an email yesterday to the SEXTANTE mailing list, pointing out the need to have some mechanism for creating your own toolbox groups and organizing algorithms, since the number of them is currently rather big due to the SAGA and GRASS interfaces. Instead of replying to his email, I have been reading the fruitful discussion that followed and implemented a first solution to that problem. And since I believe that I have done a nice work (specially considering that I have been working on this just today), I am posting this to the SEXTANTE blog to let everyone know about it. Here is what I have done:

-List of recently used algorithms. There is a group of algorithms that contains the most recently used ones and appears always at the top of the list. I might add an option to remove this group in case someone does not find it useful.

-You can arrange your algorithms as you like. Each algorithm has a provider name (Currently SEXTANTE, GRASS, Models or SAGA) and a group name. If you go to the SEXTANTE settings you will find a "configure algorithm groups" button. If you click on it the following dialog will appear:

Just change the name of the group and subgroup (the provider name and the group name) to put an algorithm on a different brach of the tree of algorithms. If you do not want an algorithm to appear, just uncheck its checkbox.

As you can see, this works even with non-native algorithms (see the SAGA label?). This is another advantage of the new architecture based on algorithm providers :-)

Groups are stored in a text file that is loaded when SEXTANTE is initialized, but I will probably implement some way of selecting your own file, so you can have several configurations. Other improvements might include sorting the algorithms table clicking on a column header or editing several algorithms in one single step. Still many things can be done, but, after all, I have just been working on that less then one full day!

All this changes have effecct both in the tollbox and in the graphical modeler.

lunes, abril 25, 2011

Big changes in SEXTANTE (II)

More changes:

-Overwriting: Until now, SEXTANTE algorithms could just generate new layers and never modify existing ones. While this is a good idea most of the times, in certain cases it must be better to just alter an input layer instead. This is the case, for instance, with the "calculate geometrical properties of polygons", which adds new fields to the attributes table of a vector layer. I have done some changes to take that into account, so algorithms now can define a link between input layers and output ones. If such a link exists, the parameter setting dialog will give the option to overwrite the corresponding input layer instead of typing an output filename.

To make this easier, I have changed the output filename panel and its behaviour. Before this, it was just a text box with a small button. Clicking on the button caused the fiel selection dialog to appear, and the textbox contained the selected filename. A blank filename forced SEXTANTE to use a temporary filename.

Now, when you click on the button, you see a more complex dialog with several tabs. In the first tab, there are two buttons: one to select a temporary output file and another one to overwrite the corresponding input, in case it is possible. The second tab is used to select a filename. More tabs can be easily added, defining different ways of storing the resulting layer. In the case of vector layer, a "database" tab appears. However, this is not functional yet, but will allow the user to select a database table as the destination for the layer to create.

The overwrite functionality is compatible not only with file-based outputs, but also with other ones, like databases. I still have to solve what happens when a layer that cannot be overwritten is selected as input, and other similar small issues. Apart from that, I think that this is also a great advance in SEXTANTE and opens many new possibilities. Once this is fully functional, I will have to adapt some algorithms, but I hope that it will not take long to do.

These functionalities (the ones detailed in this post) come mostly from the suggestions from the Deputacion de Pontevedra, who needs this kind of capabilities for their work and pointed us in the right direction (apart from funding the project that resulted in the implementation of this ideas!), so we would like to thank them for their suppport.

sábado, abril 23, 2011

Big changes in SEXTANTE

I have been making some important changes in the last 2-3 weeks and, although I still have to test them an finish a few things, most of there are already working. These changes are rather big, since they involve modification of the SEXTANTE core architecture and, specially, the sextante_gui project. Here is a detailed description of some of them.

-Removal of WPS client. The WPS client was not working and it was somehow difficult to mantain, so, after asking SEXTANTE users and developers on the mailing lists, I decided to remove it. I guess that no one was using it, and the idea was not just removing it, but reimplementing it in a different way. Removing the project was not as easy as just taking a project away, but required some changes in the core classes as well. That made me see that its design was not right, and made me think about a new way of implementing extensions to the set of native algorithms of SEXTANTE. And that's how I came up with the next big change: Algorithm Providers

-Algorihtm Providers: Along with the basic set of algorithms (those programmed in Java using the SEXTANTE base classes), SEXTANTE includes algoritms from different sources. The WPS client was one of those sources, as well as the modeler or the GRASS interface. However, all these were developed independently and needed some adjustment in the core os SEXTANTE. Although a model or an algorithm coming from GRASS is technically a SEXTANTE geoalgorithm (it extends the GeoAlgorithm class), there are some minor differences, and I solved the problems associated with those differences by adding some patches to the SEXTANTE core.

This is, however, not a very clean option, specially considering that other sources of algorithms might be added in the future. For this reason, I decided to create a mechanism for "plugging" algorithm sources in a clean and robust manner. What I have done is to create an interface named IAlgorithmProvider and methods to add ojects of this class to SEXTANTE before initializing the library, so the list of algorithms is enriched with the algorithm it provides.

An algorithm provider takes care of everything needed to incorporate algorithms, from the particular characteristics of their execution, to the location of their associated help files, and including things such as the configuration panel that the provider might need or the icon of the algorithms in the toolbox and the modeler. With this change, all algorithms behave exactly the same and there is no need to add patches to the core classes. For instance, there was a "model" command in the command line interface to execute models. Now there is just the "runalg" algorithm, and it can be used to execute any algorithm, no matter if it is a model or a GRASS algorithm or it comes from any other provider yet to be developed.

After making these changes to the core classes, I have created 3 providers. The first two ones simply adapt the modeler interface and the GRASS interface, so now they are not part of the core classes. Adapting them has been relatively easy, although the GRASS one needs some testing yet, and I have detected a few issues while rewriting it. The third provider wraps SAGA algorithms and it took me just a couple of days to develop (still need to test it in linux and add some minor stuff, but it is almost done). This is partly due to the good knowledge I have of SAGA after speding many years using it and developing many of its modules, but also due to the new SEXTANTE design, which makes it very easy to incorporate third-party algorithms through an algorithm provider. Here is a picture of the toolbox, with both SAGA and GRASS algorithms activated.

More info about other changes in the next post, to come soon...

lunes, abril 18, 2011

Reasons why I use Windows

Being an open-source programmer that uses Windows (and openly prefers it to any other operating system, Linux included) can cause you a bit of uneasiness in certain contexts. While using Windows can be frowned upon among other open-source programmer, sometimes it goes beyond that and I have to listen to endless minutes of dreadful techno-proselytism from people who, for some reason, believe that I have never opened a Linux box and that I should give it a try and that will inmediately change my whole life. Tired of hearing people asking me things like "how can you still use Windows instead of Linux?", "I do not understand how you can work with that..." and other similar ones, here is a detailed explanation of why I do what I do. Also, this is meant to serve as a reflection about open-source software and will give you some ideas about how I understand the open-source software movement and the software industry in general.

Here we go.

1)I like Windows.

This reason alone should suffice, since, basically, we all do what we like and it makes no sense to select the option that doesn't fulfill you expectations and leave aside the one that makes you happy. I spend a lot of hours in front of the computer and I prefer to do it using Windows than Linux. As simple as that. However, let me justify this with some more "professional" arguments.

As you might now, I take productivity very seriously in my work, up to the point of having no mouse or touchpad in my computer, since doing everything using the keyboard is way more productive (although it needs some training) for the kind of work that I do (that is, programming). I worked more or less one year with a Linux-only PC and my productivity was not as high as when using Windows. This is not something that I just "felt". If I was just coding, there was no difference, since I could use eclipse on both systems, but as soon as I did complex tasks involving several applications, I was less productive when using Linux. I benchmarked it using a couple of common tasks, and I concluded that, for some reason, I feel better on Windows and that it affects my productivity. It is just a matter of taste, just like some people drink coffee and others drink tea, or some listen to blues while others listen to classical music. So I am not saying that Windows is better, but just that, for me, it feels better. And I worked one year using only Linux, which I think is a long enough period of time to start comparing.

2)I run a highly tuned-up Windows.

Windows users are bad users, we all know that. They log-in with administrator priviledges and have difficulties doing common tasks. On the other hand Linux users have a great knowledge of their system, they have read a few books about it and read man pages for dozens of commands. However, not all Windows users are the same. Some of them, like myself, like to read about Windows, about how to use it properly, and with each new version of Windows check as much documentation as possible in order to find out the improvements of their operating system. And, for this reason, I can tweak my system just like any Linux user does and take the most out of it.

For instance, my computer has one administrator user but as many as five other user profiles, each of them adapted to a particular kind of work. And the "switch user" option is disabled, so changing the user requires to log off. My "working" profile makes it very easy to use eclipse but terribly hard to use any other application, so I have to concentrate on programming. The "general" user profile, on the other hand, has a one-letter shortcut for each application, so I can launch any application pressing [windows key] + R, typing one letter (c for Chrome, e for Eclipse...) and pressing Enter. The start menu is empty, so there is no way of launching apps otherwise. Another profile that I use for writing has no internet browser, since I never need that when I write, so I avoid unnecessary distractions.

Of course, you can do things like that on Linux, but how many of those saying "I cannot work on Windows" have ever taken the time to configure the system like that? Also, how many of them have the same skills on Windows as they do on Linux?. As a curious case, I can tell you of a professor that came to lecture to our university and said that he needed his Linux machine, since he liked to have several apps open along with his slides and change between them quickly, and that could be done only using several desktops in Linux, something that was not possible in Windows (he added a few adjectives here to make clear that he was convinced that Windows was a useless operating system). I was pleased to show him that the alt+tab key combination can be used to switch between open applications, and that it can be used both in Linux and Windows. He didn't know about it :-)

A fine-tuned installation of Linux is, for sure, better than an out-of-the-box installation of Windows, but not necessarily better that a properly configured Windows.

3) But Linux is free!!

I am sorry, but freedom itself is not a reason for using something. As much as I love the open-source philosophy, it is not enough to make me use something that makes me be less productive in my work and have (once angain, for me) a worse user experience. For die-hard Linux fans, that seems to be more that enough, as long as they keep fighting against the cruel, inhuman, proprietary software companies leaded by Micro$oft. Not my case, definitely.

It's funny to see how, however, most of these people buy books (thus supporting the way less-than-fair publishing companies) or records (empowering record labels and their long-ago outdated business model) while free alternatives exists. Of course, they prefer to listen to a contemporary rock band and read the latest techno-punk novel instead of downloading the complete recordings of jazz pioneer Jelly Roll Morton to their mp3-players or the complete works of Homer (written before copyright laws even existed), both of them in the public domain and freely available from a dozen of sites on the Internet. Why? Simply because they prefer their bands and writers and do not mind buying their works. Easy to understand. Now you go to point 1...

I also find it funny how we tend to think that a software distributed under a proprietary license has to be per-se a bad software. A brilliant piece of software can be distributed by a greedy company under draconian conditions, and still be technically awesome. For some reason, this seems to be difficult to understand when we speak about software, but not so much when we speak about music of literature. Metallica sued their fans for downloading their tunes using Napster, but no one (even those being sued) questioned them as probably the biggest metal band ever. They questioned them as persons, not as musicians. Similarly, no literary critic would question "Journey to the end of the night" as one of the finest novels of the XX century, although his author, Louis Ferdinand Céline, was a fascist that, along with his masterpiece, wrote quite a few anti-semitic pampleths. But, however, many people judge Windows not from a purely technical point of view after using it, but based on what they feel for Bill Gates or how much they detest the aggresive and stupid style of Steve Ballmer.

We should separate the software itself from the license, and specially, not asumme that being free makes a software better. Free software can be awful or it can be great, and it does not depend on its license but, most of all, on the programmers behind it. And, like it or not, there are awesome programmers on both sides of the proprietary vs free software world. Since this is a GIS blog, let me tell you a little secret. I get angry each time I hear about ArcGIS licenses and, specially, how the spanish distributor deals with them. Their policy is, to say it mildly, disgusting, But, regarding the software itself, I absolutely love it and would kill for having a look at its source code. The programmer in me likes to look at sofware as something with no license at all, just like a technical element not linked to any social or economical element.

I hope that this long post has helped you think a bit about this interesting topic...

Comment are welcome :-)

miércoles, abril 06, 2011


One of the main problems (IMHO) of SEXTANTE is its lack of a comprehensive user documentation. The main reason for that is that writing documentation for each single algorithm is boring (at least, more boring than developing the algorithm itself), and also that my time is limited and I prefer to do other things instead of updating the two versions of the SEXTANTE help currently available: the one in English and the one Spanish.

For this reason, I have just taken the solomonic decision of focusing only in the English documents, and not update the Spanish ones anymore. This full commitment to a single language is something that is needed to end up having a usable documentation, specially considering the small number of people working in the project.

So, in short, from now on there will be no more Spanish in SEXTANTE (except for the string in the software itself). As you can see, this decision affects not only the software, but also other elements such as this blog, which from now on will be written exclusively in english.

Hopefully, someone will help me keep the Spanish help documents (kinda) updated, but if that does not happen, at least we will have good English docs to work with. Of course, still a lot of work has to be done, but somehow I see it easier now to get somewhere, and the task doesn't seem now to me so overwhelming (which, needless to say, encourages me to work more than before)

To help people stay tuned with SEXTANTE, I will also post these entries to the users mailing list. I write just a couple of entries a month, so I guess that it will not be annoying for most of the list subscribers.

viernes, abril 01, 2011


Aquí van algunas ideas sobre los avances en los que voy trabajando últimamente y los cambios que esto supone en SEXTANTE.

1) He modificado la estructura del SVN. Ahora hay una sola carpeta (un solo proyecto) para todos los algoritmos. Esto implica ademas un solo jar con algoritmos en la distribución. Más fácil de gestionar y mucho más práctico. Voy a mantener las carpetas de los proyectos anteriores durante un tiempo, pero ya no haré cambios sobre ellas en el SVN

2)He añadido una carpeta gvSIG_2 en la carpeta de bindings del SVN. Aunque me falta un poco de trabajo, son ya unos bindings completos para SEXTANTE sobre gvSIG 2.0, que previsiblemente ha de salir pronto. Estos bindings no coinciden con los que el proyecto gvSIG tiene, y que están estructurados de forma distinta y corresponden a una versión anterior de SEXTANTE. Yo he mantenido la misma estructura que en el caso de los bindings anteriores, y el proyecto usa ant como es habitual, en lugar de maven, que es la elección del equipo de gvSIG.

3)He pulido algo los objetos de datos IVectorLayer e IRasterLayer. En el caso de IVectorLayer, he cambiado ademas el comportamiento de los iteradores, más óptimo ahora.

4)La generación de capas raster grandes ya funciona, con independencia de la plataforma. Cuando la capa es muy grande (más que un cierto umbral que estoy pensando sacar fuera como parámetro de configuración de SEXTANTE), SEXTANTE no delega en la aplicación para guardarla, sino que usa su propio sistema en disco y genera después la imagen. Por el momento, solo soporta el formato tif. Este comportamiento se podrá quitar si la aplicación tiene capacidades mayores que estas, pero por el momento creo que es lo mejor, para así poder evitar los fallos que los usuarios tienen con estas capas por falta de memoria.