miércoles, diciembre 21, 2011

Important notice for gvSIG users

Here is an email that I am going to send to the gvSIG mailing lists:

SEXTANTE and gvSIG CE teams have decided to unite forces and will be, from now on, sharing their development policies and strategies, along with some elements of their web platform (we were already using their bug tracker). Our mailing lists and SVN repository have changed, and you can check the details in our website (sextantegis.com).

This mail is to clarify how this afects SEXTANTE users and, more specifically, how future SEXTANTE releases (starting with v1.0, which will be released in a few days) will be integrated in both gvSIG CE and the official gvSIG distributions. Users of other GIS's should not worry about this. Although we will now share elements of the gvSIG CE platform, we will keep on working for other applications using SEXTANTE as we've been doing until now, and we will not abandon them.

The following points clarify the main ideas of the SEXTANTE project regarding gvSIG

-We strive to have SEXTANTE running on a large number of platforms in a robust and stable manner. Although gvSIG has been our prefered platform until now, we've found many problems in both the technical and organizational parts of the gvSIG project. This is the main reason why we are supporting the gvSIG CE project, as a better alternative for SEXTANTE users.

-Instead of keeping different versions of SEXTANTE for gvSIG and gvSIG CE, there will be only one. However, when running on the official gvSIG, some of its features will not be available, since they need some improvements made in the gvSIG CE version, which are not included in the official version. The current available gvSIG CE version is a Technical Preview and does not incorporate all those changes, but they will be included in the final stable version. As of today, there is no difference in running SEXTANTE from gvSIG CE or the official gvSIG, but changes will be important once the stable version of gvSIG CE is released. Advanced users using the current gvSIG CE SVN code can already try them.

-As it was announced in the 7th gvSIG conference this month, the gvSIG project has decided to create its own geoprocessing platform, replacing SEXTANTE. It's likely to be heavily based on SEXTANTE (it is a fork of it), but it is not the official SEXTANTE platform, so it might not include the latest changes made in SEXTANTE. We will, however, release official SEXTANTE versions specifically targeted at gvSIG users, which will be easy to install, since they will use the new gvSIG add-ons manager. SEXTANTE users will be able to install official SEXTANTE releases in gvSIG, just installing the corresponding packages that we will publish in our website. (In other words...do not worry, you will still have the good old SEXTANTE to use with whatever gvSIG version you prefer)

-Overall, we recommend all gvSIG users who use SEXTANTE to switch to gvSIG CE (once again, we're talking about the yet-to-be-released stable version, so is not an important issue right now) as a better alternative regarding SEXTANTE integration. Other features not related to SEXTANTE are, in our experience, also more reliable than those in the official version.

Thanks once again for using SEXTANTE, and we hope you enjoy our new release and keep on using SEXTANTE for your spatial analysis in gvSIG.


Victor Olaya

miércoles, noviembre 23, 2011

Kinda goodbye...

It's been almost 8 years since I decided to start the SEXTANTE project. I had an idea and I wanted to spend some time developing it and, why not, make a living from it. I ended up finding the funding I needed here in Plasencia, so I relocated from my hometown in Madrid and discovered a great city with great people, a small but charming university and the best environment I could dream of to develop what SEXTANTE is now.

I never expected SEXTANTE to grow this big and to have such an impact in my life, and I guess I can't do but thank everyone for these wonderful years.

If this sounds to you like a farewell notice...I am afraid you are right. Although I still love GIS and programming, I feel that time has come for me to do something different, and from January I will not be here anymore and I won't work full-time on SEXTANTE as I have been doing until now. I still plan to do some work, since I do not want to completely abandon SEXTANTE, so I will probably be fixing bugs and making small changes (and maybe adding some entries to this blog...). However, you cannot expect big changes or important new features to be added in the short term. At least, not done by me...but I would love to manage the SEXTANTE developers community and help them improve SEXTANTE if they want to do so.

There is still one month until all this happens, and I am working hard to release a new version with all the current changes and bug fixes. I also hope to make it easier for applications using SEXTANTE to incorporate these changes (gvSIG and gvSIG CE will both be releasing a new version soon...)

I would like to thank you all for being there and supporting SEXTANTE. This incredible time that is now ending for me would have not been the same without your support. And I hope that SEXTANTE will still be alive for a long time thanks to your collaboration, whether you are a user or a developer.

miércoles, octubre 19, 2011

Out of office

I am taking holidays tomorrow...and won't be back until the end of November. Don't worry if you do not see much activity in SEXTANTE during the following weeks. I would have loved to have more time to finish the R interface a bit more...but I will resume working on it as soon as I come back.

I usually do not check my email when I am travelling, so do not expect me to answer email or questions during these days. I hope that the last improvements of SEXTANTE can keep you all busy and having fun during this month :-)

See you all soon!


I have finaly finished a first version of the R-SEXTANTE interface. R algorithms and scripts can now be called from SEXTANTE, as is shown in the next videos:

The first one calls them from gvSIG, while the second one does it from ArcGIS. There are still a couple of additional things to be fixed in ArcGIS, and it won't take loong to do it, but I wanted to upload this notice before going home. I am pretty excited about this new development, since the possibilities of incorporating R in SEXTANTE are almos endless :-)

As you can see, there are just a few algorithms available. Unlike GRASS or SAGA, R algorithms and scripts are not incorporated automatically. Description files have to be created manually. This is very simple to do, and I expect R experts to help us with that (I am not an R expert, just have a very very basic knowledge of it). I have created a few description files as examples, and also created a small manual to document how to do it. You can find it in the documents section of the SEXTANTE website.

I hope you like this new improvement as much as I do.

jueves, septiembre 15, 2011

About the SEXTANTE support for 3rd party algorithm providers

I have been receiving many emails lately about the support for SAGA and GRASS in SEXTANTE, how to configure it and what is our policy about that. Also, I am helping a colleague to create the custom distribution for the SEXTANTE course organized by the gvSIG Association that he will teach in October, and a few similar issues have arisen. For this reason, I am writing this entry to clarify all this, hoping that it will help users and integrators of SEXTANTE.

The main idea is that, due to inestability problems, we have decided that the installation process has to be done in advance and cannot be done by the user. Letting the user do this is error-prone, since different versions can be used and there are many others things than have to be controlled. We believe that this in not good for SEXTANTE, so we will support only preconfigured systems.

For this reason, the SAGA and GRASS providers will only be available if the application includes SAGA and GRASS and they are correctly preconfigured.

So far, this is the situation regarding the main GIS software that uses SEXTANTE:

-gvSIG CE will ship with both and SAGA and GRASS, and will be thus complete with all algorithms.

-Official gvSIG will ship (AFAIK) without them, and will have no support for SAGA and GRASS. This might, however, change, since we are working with them to prepare for their next release and we haven't discussed that carefully. Right now, there are two different bindings for these two flavors of gvSIG, and the SAGA/GRASS support is the main difference between them (though not the only one)

-SEXTANTE for ArcGIS will include a running version of SAGA (Windows only), but no GRASS, since we are not completely happy with the behaviour of the SEXTANTE/GRASS interface when running under ArcGIS. This distribution can already be downloaded from our website (still a beta, not a final release).

-OpenJUMP will have no support for external providers (mainly due to our lack of time to work on it...sorry)

martes, agosto 30, 2011

SEXTANTE algorithms in the ArcGIS Model Builder

This new video shows how SEXTANTE algorithms can run in the ArcGIS Model Builder. The model is a very simple one, but executes a SAGA algorithm (convergence index) and a native SEXTANTE one (Histogram). Since the result is a histogram chart, it has to be viewed using the SEXTANTE results window, which is now available from an add-in that I developed, which adds a button to open it.

SAGA and GRASS running in ArcGIS through SEXTANTE

Here is another video of SEXTANTE on ArcGIS. This time, I am showing how SAGA and SEXTANTE algorithms can be executed just like native SEXTANTE algorithms. This is completely true for SAGA ones, but I'm having a bit of trouble with GRASS ones, specially vector ones. This is not a GRASS or SEXTANTE problem, but just a problem caused by the strange behaviour of Java when I call the application (SAGA or GRASS) from ArcGIS, since it freezes and I haven't been able to solve it. If I execute a new interpreter (that's why the command line window is shown), it works but it returns the control back to the Java code before the task is finished. All this doesn't happen when calling the, from a pure java application, like OJ or gvSIG, so I have had to rewrite part of the GRASS interface in SEXTANTE. The SAGA one is simpler, so just a minor tweak was needed.

Anyway, I will keep on working on this, but for now SAGA algorithms are fully available, and I hope that GRASS ones will soon be as well.

Both native SEXTANTE algorithms and SAGA/GRASS ones also work within the Model Builder. I am preparing a video about that that I will post soon.

ArcGIS coupled with SEXTANTE SAGA and GRASS is really a powerful geospatial analysis app, bringing together the best of both worlds: propietary and open source. Needless to say, it makes me really happy to see that SEXTANTE is the bonding element responsible for that.

martes, agosto 16, 2011

First video of SEXTANTE for ArcGIS

Here is a first video of SEXTANTE for ArcGIS, so you can see what it is like to access the power of SEXTANTE from the popular ESRI product. I recommend to watch full screen or at least at a larger size then the embedded video, to fully see what is going on in the screen...

As you can see, output results are not added automatically to the view (I haven't figured out how to do it yet...and this is the one single thing that I can't find documentation about...Anyone can help??), but the remaining elements of the SEXTANTE-ArcGIS interaction are already implemented and seem to be working pretty fine.

I still want to do some more testing and solve a few minor issues, but everything is going fine and the next version of SEXTANTE will for sure ship with fully-working ArcGIS bindings

We are all very excited about this development, since it means a big step in the (short but intense) history of SEXTANTE :-)

lunes, julio 18, 2011

Adding external functions from gvSIG to SEXTANTE, and more.

The folks at the gvSIG Community Edition project are doing an awesome work trying to create a better version of gvSIG, and they are putting a lot of effort in trying to enhance the integration of SEXTANTE. After 3 days of fruitful and really fun work (met with them at AGIT, in Salzburg), we believe that gvSIG CE is going to become the best platform for using SEXTANTE and, thus, support their work and have given some advise about how to better integrate SEXTANTE with their software. Also, I have made a few changes that will greatly improve the experience of using SEXTANTE and will give gvSIG CE users new posibilities. The most important are the two following ones:

1)Introduction of Toolbox Actions. A toolbox action is just a command that you can call from the toolbox, so now you can have not only SEXTANTE algorithms but any other thing that you want. Of course, this actions cannot be used in other elements of SEXTANTE like the modeler, but it is handy to have some functionalities close to your geoalgorithms. Why is this useful for gvSIG CE? Well, we all now that the current distributions of icons and menus is a mess in gvSIG. There is a geoprocessing toolbox with just vector algorithms, then there is a network analysis interface and finally there are some raster functions that you can access from an awkward double-drop-down-list system in the toolbar. Having all of them together would make the interface much more homogeneous and easy to use, so the idea is to create actions to call those elements and then remove their corresponding menus and icons. Everything related to spatial analysis will be found in the SEXTANTE toolbox.

The idea of having just one geoprocessing framework has been around in gvSIG since many years ago, but it is easy to see that using that framework hasn't been really enforced by the gvSIG team. Of course, it would be better to have all those functions as real SEXTANTE algorithms. Although we plan to do so, that means a lot of work, so this solutions is good by now until we have a better one.

2) Changes for a portable version. The first version of gvSIG CE will be a portable one (no need to install anything), bundled with GRASS and SAGA and fully preconfigured. That means that you will have more than 700 algorithms from the very first time you run the software and there will be no need to configure anything about SAGA or GRASS. This wasn't possible before, since configuration files were kept in a user folder, and thus it was not very portable.

Also, Benjamin Ducke, the frontman of the gvSIG CE version, is responsible of developing most of the GRASS interface in SEXTANTE, so we worked together on that and made a few changes. We both agree in that a bundled installation of GRASS or SAGA is prefered to an external one, and we made changes in that direction, in order to have a more robust integration assuming that the user has a preinstalled copy of any of these addtional softwares.

The fist version of gvSIG CE will be out soon (its release will probably be synchronized with the release of SEXTANTE 0.7) and we highly recommend everyone to try it, specially if you work with SEXTANTE. As I said, we see this version as the ultimate SEXTANTE platform, since it has been specifically adapted and solves some important issues.


viernes, junio 24, 2011

Is propietary software better for developers?

I have recently been programming the SEXTANTE bindings for ArcGIS (they are not finished yet, since I am having a very busy week with other priorities, but I will resume working on them next week and hope to have them ready soon) and working with the ArcGIS API has made me think a bit about how the propietary software model can somehow be an advantage for a certain kind of developers. I am not, of course, saying that propietary means better (I will always prefer open source to propietary), but there might be a few things that we can learn from propietary software developers and some positive side effects of not sharing your source code.

This all comes from the following fact: being a complete ignorant about ArcObjects and the ArcGIS API, I have been able to create the corresponding SEXTANTE bindings from scratch in less time than I needed to create any other bindings before. That means less time than the gvSIG bindings (an application that I knew pretty well), and way less than the OpenJUMP or Geotools ones (both of them softwares that I had worked with before, at least once).

This is basically due to the clean, well designed and perfectly documented API of ArcGIS, which, along with the additional Eclipse plugins, makes it very easy to develop new plugins and extensions for the software. Unfortunately, open-source GIS lack that kind of APIs, or they are, at least, not so carefully designed, documented and implemented. Although other factors might concur, I am convinced that the propietary nature of ArcGIS is a decisive one to have such a powerful and developer-friendly API.

Let's put it this way: ArcGIS developers want other developers to create plugins for their software, so they have to provide an easy way of doing so. However, they cannot let you see the source code or their application, so they have to make sure that everything that an external developer might need to create his plugin is correctly exposed through their API. The API has to, somehow, replace the source code for certain development tasks. On the other hand, open-source programmers are aware of the importance of developing a correct and well documented API, but the availability of source-code might act as an excuse for sloppy or unfinished APIs, assuming that, if an external developers misses anything in it will search the source code and try to find whatever he is missing.

That doesn't necesarily mean that open-source APIs are not as good as proprietary ones, but, while an open-source software, even if it lacks documentation and has a poorly designed API, it is still useful (albeit frustrating) for a developer, the same circumstances will render a proprietary software useless in term of development. That alone should encourage proprietary software developers to create and document practical and useful API (and we all know that this is important, since that is not always a nice and rewarding task...)

Once again, I insist that I am not praising the proprietary model, but just trying to find its benefits and trying to learn from it and apply it later to my open-source projects. I believe that everyone can teach us something and there is always something to learn from everyone. And, although I am not one of those who see proprietary software as an enemy, even enemies can teach you a few valuable lessons and it is not a good idea to ignore them. Quoting the much-missed Rage Against the Machine: Know your enemy :-)

miércoles, junio 15, 2011

Bug Tracker

I am pleased to announce that we already have a fully-working SEXTANTE bug tracker. Instead of using OSOR's one, we are now sharing the gvSIG CE (Community Edition) tracker. I will soon put a link on our website. In the meantime, here is the tracker URL:


This also shows our commitment to help this interesting initiative, and we will probably use the Wiki and forums of the gvSIG CE project, hoping it might help to unite all SEXTANTE/gvSIG users and work together to improve both software packages.

lunes, junio 06, 2011

Scripts as algorithms

i have been improving the command-line interface and I have added one new feature that I guess will be interesting for SEXTANTE power users.

The command line interface allows the creation of scripts that can be put in a separate file and then executed with the source(filename) command. If a method is written to to that file, then executing the file will make the method available in the current command-line session.

For instance, if we have a method like the next one,

mymethod(rasterLayer, vectorLayer){
//do something with those layers

executing the corresponding file will add a new method calle "mymethod" that I can use anytime.

Since we are running a scripting language, there is no need to specify data types for method parameters. However, now, if you add some information about those types as comments before the method, SEXTANTE can not just execute them from the command line, but also from the toolbox as a normal algorithm.

The above method should be something like this:

mymethod(rasterLayer, vectorLayer){
//do something with those layer

Those two comments are used by SEXTANTE to create the corresponding dialog.

If you put the bsh file in the scripts folder (which now you can define in the settings dialog), the method it contains will appear automatically in the toolbox as any other algorithm or model.

I still have to work more on this and document it, but I guess it's a nice beginning for a rather intereesting idea.

martes, mayo 31, 2011

some little improvements

Most of my work in this last days has been solving minor bugs and adding small improvements to the SEXTANTE GUI. Here is a quick review of some of them. Not all of them have been finished and tested yet.

-Optional outputs. There is a new output channel (NullOutputChannel) that basically does nothing with the result of an algorithm. It can be selected from the output channel selection dialog, and doing so causes SEXTANTE not to create the corresponding output layer. This was, you can have optional outputs, but there is no need to set them as optional in the source code of each algorithm o take that into account somehow.

-GRASS algorithms in modeler. GRASS algorithms work now in the modeler, and I have already implemented a mechanism for dealing with the major issue: layers with more than one geometry type. When you execute a GRASS algorithm now, in case it generates vector layers with muliple geometry types, you have to select the type of geometry you want to use (the other ones are discarded). The selection is already implemented, but the GrassAlgorithm class is not using it yet. I will code that in a month or so, when I go to Salzburg to atend the AGIT conference, since we plan to have a code sprint there with other developers who know more than me about GRASS...

-Definition of analysis extent using stored points. Point stored using the catch coordinates tool can now be used to manually define the extent of the analysis region.

miércoles, mayo 11, 2011

New website and video tutorials

Maybe you have already realized that I have changed the sextantegis.com website. Instead of redirecting to the OSOR wiki, now it redirects to a minimalistic nice-looking web page, which is, IMHO, much better and more practical.

If you go to the website, you will also see that there is a link to our brand new YouTube channel. I have already uploaded 13 video tutorials (around 3 min each), and plan to add many more in the following days (I try to do 1 or 2 each day...). I am recording them in english, in order to make them useful for everyone (I absolutely do not like my accent, but I hope you find them interesting anyway ;-) ).

Here is a quick list of all the ones that I want to have ready soon:

-the toolbox
-the parameters window
-the output region tab
-output channels
-capturing points
-the graphical modeler
-modeler inputs
-modeler inputs(II)
-modeler algorithms
-saving and configuring models
-batch processing
-batch processing(II)
-batch processing from open layers
-Numerical results
-Iterative execution of algorithms
-Basic operations with raster layers
-Focal statistics
-Local statistics
-The raster calculator
-Using masks
-Overlay operations with vector layers
-Dissolving vector layers
-Basic spatial statistics with vector layers
-Basic transformations for vector layers
-Basic transformations for vector layers(II)
-Creating profiles
-Creating a raster layer from a set of points
-Creating a density layer

Many more can be added to that list (and you can be sure that I will do it), but I would like to have your opinion. Is there anything that you would like to see there? Any process that you would like me to document in a video? If so, please feel free to tell me. I think video tutorials are great and easy to make, so all suggestions are welcome.

lunes, mayo 02, 2011

On ultrarunning, GIS and programming on the run.

Those of you that do not know me personally probably do not know that one of my biggest passions is running. And when I mean running, I mean running *a lot*. I have been into endurance sports since I was a kid, and since a few years ago I am a passionate ultrarunner, that meaning that I enjoy a 20 km run, but I have real fun when I get out of home early in the morning and come back home several hours later in the evening with 60-70 km [~43 miles for those not used to the International System] on my legs.

To run such long distances, a fair amount of physical endurance is needed but, more that that, a great psychological endurance is the key. As the classic ultrarunning mantra goes: "pain is inevitable, suffering is optional", and each ultrarunner has his own strategy for ignoring the pain and turning a long run into a rewarding (albeit painful) journey. Some like to listen to music, others simply think about their life. Dean Karnazes, by far the most popular ultrarunner around, wrote most of his best-seller "Ultramarathon man, confessions of an all-night runner" while running, recording his thoughts into a portable recorder and later transcribing them.

Now you must be wondering why I am writing this in this blog about GIS, SEXTANTE and programming. Well, there is a link between SEXTANTE and running, and I would even dare to say that SEXTANTE would not be the same if it were not for my long weekend runs. I do not like to listen to music while I run, and I definitely do not feel like bringing a recorder with me when I go running, but there is something that I love to do when I run. Guess it? That's right: programming. (Another thing that I usually do is to practice scales...but I have a blog about music to talk about that, so I will not bore you here with the details ;-) )

As the great Edsger Dijkstra said, "computer science is not more about computers than astronomy is about telescopes". In other words, you do not need a computer to program. Of course I do not do actual coding while running, but programming is much more that just typing code. Design and optimization are two of the things that can be done without having a computer in front of you. In fact, I have realized that I tend to code too much and think less than I should if I am sitting in front of a computer, so it is better for me to spend sometime "working" without a computer. Staying away from the computer lets me concentrate on details that I might overlook if I am typing code.

Also, running for several hours to the edge of exhaustion gives you an extremely clear thought once you get past the point where your mind and your body separate (and believe me, this happens sooner or later in every long run...otherwise your body would say no and stop), so, somehow, you get "cleverer" as you run. That makes it an ideal situation for thinking about such abstract problems, and for finding optimal solutions to them.

I am writing this here because, as I mentioned in some of my latest posts, there have been some important new functionalities added to SEXTANTE in the last weeks. Most of these new functionalities involve changes in the architecture of SEXTANTE and, thus, require not just coding, but also rethinking part of its design. And, as you might have probably figured out, I did a big part of that while running. The feeling of arriving home, eating something, having a shower (or, better, a freezing ice bath) and sitting in front of the computer to put your ideas to practice is just great, letting your thoughts flow and turn into code. Definitely, much better that sitting in my office at the university (although I usually go for a short early morning 10-15km run before going to work), and by far much more productive.

Now a question to all programmers around: Do you have any preferred moment of the day to work? Anything you like to do to get ready or boost your productivity and/or creativity?

By the way, I thought about this post and "wrote" most of it a few days ago on my mind during a long and beautiful 190+ km (half of them under pouring rain...) bike ride on the almost-abandoned roads around my town. Also, whenever I have a presentation or talk, I usually rehearse it while on the run, so I guess that SEXTANTE owes more than just code to my running/cycling sessions :-)

jueves, abril 28, 2011

Organizing your algorithms

Benjamin Ducke posted an email yesterday to the SEXTANTE mailing list, pointing out the need to have some mechanism for creating your own toolbox groups and organizing algorithms, since the number of them is currently rather big due to the SAGA and GRASS interfaces. Instead of replying to his email, I have been reading the fruitful discussion that followed and implemented a first solution to that problem. And since I believe that I have done a nice work (specially considering that I have been working on this just today), I am posting this to the SEXTANTE blog to let everyone know about it. Here is what I have done:

-List of recently used algorithms. There is a group of algorithms that contains the most recently used ones and appears always at the top of the list. I might add an option to remove this group in case someone does not find it useful.

-You can arrange your algorithms as you like. Each algorithm has a provider name (Currently SEXTANTE, GRASS, Models or SAGA) and a group name. If you go to the SEXTANTE settings you will find a "configure algorithm groups" button. If you click on it the following dialog will appear:

Just change the name of the group and subgroup (the provider name and the group name) to put an algorithm on a different brach of the tree of algorithms. If you do not want an algorithm to appear, just uncheck its checkbox.

As you can see, this works even with non-native algorithms (see the SAGA label?). This is another advantage of the new architecture based on algorithm providers :-)

Groups are stored in a text file that is loaded when SEXTANTE is initialized, but I will probably implement some way of selecting your own file, so you can have several configurations. Other improvements might include sorting the algorithms table clicking on a column header or editing several algorithms in one single step. Still many things can be done, but, after all, I have just been working on that less then one full day!

All this changes have effecct both in the tollbox and in the graphical modeler.

lunes, abril 25, 2011

Big changes in SEXTANTE (II)

More changes:

-Overwriting: Until now, SEXTANTE algorithms could just generate new layers and never modify existing ones. While this is a good idea most of the times, in certain cases it must be better to just alter an input layer instead. This is the case, for instance, with the "calculate geometrical properties of polygons", which adds new fields to the attributes table of a vector layer. I have done some changes to take that into account, so algorithms now can define a link between input layers and output ones. If such a link exists, the parameter setting dialog will give the option to overwrite the corresponding input layer instead of typing an output filename.

To make this easier, I have changed the output filename panel and its behaviour. Before this, it was just a text box with a small button. Clicking on the button caused the fiel selection dialog to appear, and the textbox contained the selected filename. A blank filename forced SEXTANTE to use a temporary filename.

Now, when you click on the button, you see a more complex dialog with several tabs. In the first tab, there are two buttons: one to select a temporary output file and another one to overwrite the corresponding input, in case it is possible. The second tab is used to select a filename. More tabs can be easily added, defining different ways of storing the resulting layer. In the case of vector layer, a "database" tab appears. However, this is not functional yet, but will allow the user to select a database table as the destination for the layer to create.

The overwrite functionality is compatible not only with file-based outputs, but also with other ones, like databases. I still have to solve what happens when a layer that cannot be overwritten is selected as input, and other similar small issues. Apart from that, I think that this is also a great advance in SEXTANTE and opens many new possibilities. Once this is fully functional, I will have to adapt some algorithms, but I hope that it will not take long to do.

These functionalities (the ones detailed in this post) come mostly from the suggestions from the Deputacion de Pontevedra, who needs this kind of capabilities for their work and pointed us in the right direction (apart from funding the project that resulted in the implementation of this ideas!), so we would like to thank them for their suppport.

sábado, abril 23, 2011

Big changes in SEXTANTE

I have been making some important changes in the last 2-3 weeks and, although I still have to test them an finish a few things, most of there are already working. These changes are rather big, since they involve modification of the SEXTANTE core architecture and, specially, the sextante_gui project. Here is a detailed description of some of them.

-Removal of WPS client. The WPS client was not working and it was somehow difficult to mantain, so, after asking SEXTANTE users and developers on the mailing lists, I decided to remove it. I guess that no one was using it, and the idea was not just removing it, but reimplementing it in a different way. Removing the project was not as easy as just taking a project away, but required some changes in the core classes as well. That made me see that its design was not right, and made me think about a new way of implementing extensions to the set of native algorithms of SEXTANTE. And that's how I came up with the next big change: Algorithm Providers

-Algorihtm Providers: Along with the basic set of algorithms (those programmed in Java using the SEXTANTE base classes), SEXTANTE includes algoritms from different sources. The WPS client was one of those sources, as well as the modeler or the GRASS interface. However, all these were developed independently and needed some adjustment in the core os SEXTANTE. Although a model or an algorithm coming from GRASS is technically a SEXTANTE geoalgorithm (it extends the GeoAlgorithm class), there are some minor differences, and I solved the problems associated with those differences by adding some patches to the SEXTANTE core.

This is, however, not a very clean option, specially considering that other sources of algorithms might be added in the future. For this reason, I decided to create a mechanism for "plugging" algorithm sources in a clean and robust manner. What I have done is to create an interface named IAlgorithmProvider and methods to add ojects of this class to SEXTANTE before initializing the library, so the list of algorithms is enriched with the algorithm it provides.

An algorithm provider takes care of everything needed to incorporate algorithms, from the particular characteristics of their execution, to the location of their associated help files, and including things such as the configuration panel that the provider might need or the icon of the algorithms in the toolbox and the modeler. With this change, all algorithms behave exactly the same and there is no need to add patches to the core classes. For instance, there was a "model" command in the command line interface to execute models. Now there is just the "runalg" algorithm, and it can be used to execute any algorithm, no matter if it is a model or a GRASS algorithm or it comes from any other provider yet to be developed.

After making these changes to the core classes, I have created 3 providers. The first two ones simply adapt the modeler interface and the GRASS interface, so now they are not part of the core classes. Adapting them has been relatively easy, although the GRASS one needs some testing yet, and I have detected a few issues while rewriting it. The third provider wraps SAGA algorithms and it took me just a couple of days to develop (still need to test it in linux and add some minor stuff, but it is almost done). This is partly due to the good knowledge I have of SAGA after speding many years using it and developing many of its modules, but also due to the new SEXTANTE design, which makes it very easy to incorporate third-party algorithms through an algorithm provider. Here is a picture of the toolbox, with both SAGA and GRASS algorithms activated.

More info about other changes in the next post, to come soon...

lunes, abril 18, 2011

Reasons why I use Windows

Being an open-source programmer that uses Windows (and openly prefers it to any other operating system, Linux included) can cause you a bit of uneasiness in certain contexts. While using Windows can be frowned upon among other open-source programmer, sometimes it goes beyond that and I have to listen to endless minutes of dreadful techno-proselytism from people who, for some reason, believe that I have never opened a Linux box and that I should give it a try and that will inmediately change my whole life. Tired of hearing people asking me things like "how can you still use Windows instead of Linux?", "I do not understand how you can work with that..." and other similar ones, here is a detailed explanation of why I do what I do. Also, this is meant to serve as a reflection about open-source software and will give you some ideas about how I understand the open-source software movement and the software industry in general.

Here we go.

1)I like Windows.

This reason alone should suffice, since, basically, we all do what we like and it makes no sense to select the option that doesn't fulfill you expectations and leave aside the one that makes you happy. I spend a lot of hours in front of the computer and I prefer to do it using Windows than Linux. As simple as that. However, let me justify this with some more "professional" arguments.

As you might now, I take productivity very seriously in my work, up to the point of having no mouse or touchpad in my computer, since doing everything using the keyboard is way more productive (although it needs some training) for the kind of work that I do (that is, programming). I worked more or less one year with a Linux-only PC and my productivity was not as high as when using Windows. This is not something that I just "felt". If I was just coding, there was no difference, since I could use eclipse on both systems, but as soon as I did complex tasks involving several applications, I was less productive when using Linux. I benchmarked it using a couple of common tasks, and I concluded that, for some reason, I feel better on Windows and that it affects my productivity. It is just a matter of taste, just like some people drink coffee and others drink tea, or some listen to blues while others listen to classical music. So I am not saying that Windows is better, but just that, for me, it feels better. And I worked one year using only Linux, which I think is a long enough period of time to start comparing.

2)I run a highly tuned-up Windows.

Windows users are bad users, we all know that. They log-in with administrator priviledges and have difficulties doing common tasks. On the other hand Linux users have a great knowledge of their system, they have read a few books about it and read man pages for dozens of commands. However, not all Windows users are the same. Some of them, like myself, like to read about Windows, about how to use it properly, and with each new version of Windows check as much documentation as possible in order to find out the improvements of their operating system. And, for this reason, I can tweak my system just like any Linux user does and take the most out of it.

For instance, my computer has one administrator user but as many as five other user profiles, each of them adapted to a particular kind of work. And the "switch user" option is disabled, so changing the user requires to log off. My "working" profile makes it very easy to use eclipse but terribly hard to use any other application, so I have to concentrate on programming. The "general" user profile, on the other hand, has a one-letter shortcut for each application, so I can launch any application pressing [windows key] + R, typing one letter (c for Chrome, e for Eclipse...) and pressing Enter. The start menu is empty, so there is no way of launching apps otherwise. Another profile that I use for writing has no internet browser, since I never need that when I write, so I avoid unnecessary distractions.

Of course, you can do things like that on Linux, but how many of those saying "I cannot work on Windows" have ever taken the time to configure the system like that? Also, how many of them have the same skills on Windows as they do on Linux?. As a curious case, I can tell you of a professor that came to lecture to our university and said that he needed his Linux machine, since he liked to have several apps open along with his slides and change between them quickly, and that could be done only using several desktops in Linux, something that was not possible in Windows (he added a few adjectives here to make clear that he was convinced that Windows was a useless operating system). I was pleased to show him that the alt+tab key combination can be used to switch between open applications, and that it can be used both in Linux and Windows. He didn't know about it :-)

A fine-tuned installation of Linux is, for sure, better than an out-of-the-box installation of Windows, but not necessarily better that a properly configured Windows.

3) But Linux is free!!

I am sorry, but freedom itself is not a reason for using something. As much as I love the open-source philosophy, it is not enough to make me use something that makes me be less productive in my work and have (once angain, for me) a worse user experience. For die-hard Linux fans, that seems to be more that enough, as long as they keep fighting against the cruel, inhuman, proprietary software companies leaded by Micro$oft. Not my case, definitely.

It's funny to see how, however, most of these people buy books (thus supporting the way less-than-fair publishing companies) or records (empowering record labels and their long-ago outdated business model) while free alternatives exists. Of course, they prefer to listen to a contemporary rock band and read the latest techno-punk novel instead of downloading the complete recordings of jazz pioneer Jelly Roll Morton to their mp3-players or the complete works of Homer (written before copyright laws even existed), both of them in the public domain and freely available from a dozen of sites on the Internet. Why? Simply because they prefer their bands and writers and do not mind buying their works. Easy to understand. Now you go to point 1...

I also find it funny how we tend to think that a software distributed under a proprietary license has to be per-se a bad software. A brilliant piece of software can be distributed by a greedy company under draconian conditions, and still be technically awesome. For some reason, this seems to be difficult to understand when we speak about software, but not so much when we speak about music of literature. Metallica sued their fans for downloading their tunes using Napster, but no one (even those being sued) questioned them as probably the biggest metal band ever. They questioned them as persons, not as musicians. Similarly, no literary critic would question "Journey to the end of the night" as one of the finest novels of the XX century, although his author, Louis Ferdinand Céline, was a fascist that, along with his masterpiece, wrote quite a few anti-semitic pampleths. But, however, many people judge Windows not from a purely technical point of view after using it, but based on what they feel for Bill Gates or how much they detest the aggresive and stupid style of Steve Ballmer.

We should separate the software itself from the license, and specially, not asumme that being free makes a software better. Free software can be awful or it can be great, and it does not depend on its license but, most of all, on the programmers behind it. And, like it or not, there are awesome programmers on both sides of the proprietary vs free software world. Since this is a GIS blog, let me tell you a little secret. I get angry each time I hear about ArcGIS licenses and, specially, how the spanish distributor deals with them. Their policy is, to say it mildly, disgusting, But, regarding the software itself, I absolutely love it and would kill for having a look at its source code. The programmer in me likes to look at sofware as something with no license at all, just like a technical element not linked to any social or economical element.

I hope that this long post has helped you think a bit about this interesting topic...

Comment are welcome :-)

miércoles, abril 06, 2011


One of the main problems (IMHO) of SEXTANTE is its lack of a comprehensive user documentation. The main reason for that is that writing documentation for each single algorithm is boring (at least, more boring than developing the algorithm itself), and also that my time is limited and I prefer to do other things instead of updating the two versions of the SEXTANTE help currently available: the one in English and the one Spanish.

For this reason, I have just taken the solomonic decision of focusing only in the English documents, and not update the Spanish ones anymore. This full commitment to a single language is something that is needed to end up having a usable documentation, specially considering the small number of people working in the project.

So, in short, from now on there will be no more Spanish in SEXTANTE (except for the string in the software itself). As you can see, this decision affects not only the software, but also other elements such as this blog, which from now on will be written exclusively in english.

Hopefully, someone will help me keep the Spanish help documents (kinda) updated, but if that does not happen, at least we will have good English docs to work with. Of course, still a lot of work has to be done, but somehow I see it easier now to get somewhere, and the task doesn't seem now to me so overwhelming (which, needless to say, encourages me to work more than before)

To help people stay tuned with SEXTANTE, I will also post these entries to the users mailing list. I write just a couple of entries a month, so I guess that it will not be annoying for most of the list subscribers.

viernes, abril 01, 2011


Aquí van algunas ideas sobre los avances en los que voy trabajando últimamente y los cambios que esto supone en SEXTANTE.

1) He modificado la estructura del SVN. Ahora hay una sola carpeta (un solo proyecto) para todos los algoritmos. Esto implica ademas un solo jar con algoritmos en la distribución. Más fácil de gestionar y mucho más práctico. Voy a mantener las carpetas de los proyectos anteriores durante un tiempo, pero ya no haré cambios sobre ellas en el SVN

2)He añadido una carpeta gvSIG_2 en la carpeta de bindings del SVN. Aunque me falta un poco de trabajo, son ya unos bindings completos para SEXTANTE sobre gvSIG 2.0, que previsiblemente ha de salir pronto. Estos bindings no coinciden con los que el proyecto gvSIG tiene, y que están estructurados de forma distinta y corresponden a una versión anterior de SEXTANTE. Yo he mantenido la misma estructura que en el caso de los bindings anteriores, y el proyecto usa ant como es habitual, en lugar de maven, que es la elección del equipo de gvSIG.

3)He pulido algo los objetos de datos IVectorLayer e IRasterLayer. En el caso de IVectorLayer, he cambiado ademas el comportamiento de los iteradores, más óptimo ahora.

4)La generación de capas raster grandes ya funciona, con independencia de la plataforma. Cuando la capa es muy grande (más que un cierto umbral que estoy pensando sacar fuera como parámetro de configuración de SEXTANTE), SEXTANTE no delega en la aplicación para guardarla, sino que usa su propio sistema en disco y genera después la imagen. Por el momento, solo soporta el formato tif. Este comportamiento se podrá quitar si la aplicación tiene capacidades mayores que estas, pero por el momento creo que es lo mejor, para así poder evitar los fallos que los usuarios tienen con estas capas por falta de memoria.

martes, marzo 15, 2011

Nuevas funcionalidades para gvSIG-EIEL

La Diputación de Pontevedra nos encargó en su momento la realización de algunos algoritmos de análisis nuevos para incorporar a su aplicación para la EIEL (gvSIG-EIEL). Además de añadir estos algoritmos, se creó una interfaz específica para ellos, de forma que se pudieran acceder de manera personalizada y no a través de la caja de herramientas clásica de SEXTANTE y los cuadros de diálogo automáticos que se generan desde esta. No obstante, la base sobre la que se asientan los algoritmos es SEXTANTE, con lo que los éstos pueden emplearse de la forma habitual. Algunos de ellos (los que presentan un interés más genérico), están ya añadidos en el repositorio de SEXTANTE y se publicarán en la próxima versión, entre ellos algunos de gran utilidad tales como los correspondientes a enlaces espaciales de diversos tipos. En posteriores entradas comentaré más al respecto sobre estos algoritmos.

Es de reseñar el apoyo al software libre por parte de la Diputación, y el hecho de que SEXTANTE responde como plataforma a las necesidades de distintos usuarios, que se apoyan en ella para implementar sus formulaciones particulares.

Actualmente la Diputación de Pontevedra ha optado por emplear el proyecto oficial sectorial de la Asociación gvSIG para la gestión de la Encuesta de Infraestructura y Equipamiento Local, denominado gvSIG-EIEL (que supone una evolución de OpenGIS-EIEL), y sobre el que se están incorporando también estas nuevas funcionalidades

lunes, febrero 14, 2011

Salidas numéricas

Llevo un tiempo sin escribir en el blog y se me han acumulado las cosas que contar, así que estos días voy a intentar ponerme al día y contar las novedades que he desarrollado últimamente, que no son pocas. Una de las más destacables es la incorporación de resultados numéricos desde un algoritmo de SEXTANTE.

Antes de esto, había algoritmos que generaban resultados numéricos (por ejemplo, estadísticas), aunque estos se presentaban como un resultado de texto, debidamente formateados. Eso permitía al usuario usar esos resultados, pero, al no existir semántico de ellos, no permitía que fuera el programa quien los utilizara para esas cosas. Ahora, al crear un algoritmo, se pueden declarar salidas numéricas que, aunque no se muestran (salvo en un caso particular que ahora veremos), están disponibles para hacer otras cosas con ellas. Esto permite, por ejemplo:

  • Usar salidas numéricas de un algoritmo como entradas de otro en el modelizador. He añadido ademas un nuevo algoritmo llamado "Calculadora", que permite modificar esos valores mediante operaciones aritméticas, de forma que puede hacerse que un algoritmo tome en uno de sus parámetros el valor de la media de una capa raster elevado al cuadrado, por ejemplo. Estoy en proceso de modificar la calculadora de mapas y la calculadora de campos para que puedan meterse esos valores como parte de las fórmulas.

  • Mejor estructuración de las salidas en ejecuciones múltiples. Si se ejecuta un algoritmo que genere alguno de esos textos con valores numéricos de forma iterativa, esto puede dar lugar a un número muy elevado de tales resultados, difíciles de aprovechar después. En ese caso, SEXTANTE ahora prepara dichos datos en una tabla, de forma que son más sencillos de analizar y más manejables.

domingo, febrero 13, 2011

Ejecución iterativa de algoritmos.

Una novedad que llevará la próxima versión de SEXTANTE (y que ya está disponible en las versiones diarias desde hace algún tiempo e incluso la presenté en las jornadas de gvSIG en diciembre) es la ejecución iterativa de algoritmos. Para todos aquellos algoritmos que toman como entrada alguna capa vectorial, es posible ejecutar dicho algoritmo sobre cada una de las entidades de esa capa. Es decir, que si, por ejemplo, contamos con una capa de polígonos con 10 entidades y ejecutamos el algoritmo "Cortar capa raster con capa de polígonos", al hacerlo iterativamente se ejecutará ese algoritmo 10 veces, tomando cada vez como entrada vectorial una capa que contiene un único polígono. De este modo se generarán 10 nuevas capas raster, automatizándose el proceso notablemente.

Un resultado similar se podría obtener anteriormente separando la capa vectorial en n capas vectoriales de una entidad y después recurriendo al proceso por lotes. Ahora SEXTANTE permite hacer eso en un único paso.

Para acceder a la ejecución iterativa de un algoritmo, basta pinchar sobre su nombre en la caja de herramientas. Aparecerán tantas opciones de ejecución vectorial como parámetros de tipo capa vectorial requiera el algoritmo, ya que no es posible iterar sobre varias capas a la vez, debiendo elegirse una de ellas en caso de que el algoritmo necesite varias.

Para más detalles, puede verse el video de la presentación de gvSIG que he mencionado, que está ya disponible aquí.

lunes, enero 10, 2011

Sobre la Ley Sinde, los creadores y otras historias.

En estos tiempos en que la Ley Sinde está en boca de todos y en buen número de ocasiones para decir bastantes tonterías, quiero aportar mi granito de arena a la discusión desde este blog. Tratándose del blog de un software libre, mi postura al respecto no debiera sorprender a nadie, pero aún así creo que siempre hay formas distintas de matizar o argumentar lo que uno piensa.

La Ley Sinde admite muchas interpretaciones y aquellos que la defienden esgrimen muchos argumentos para justificarla. Sin atreverme a afirmar que todos ellos son incorrectos, hay uno de ellos que considero especialmente falaz y me irrita en particular, y que veo últimamente demasiado a menudo: sin una ley como esta, y dejando el mundo de la creación en las crueles manos de "piratas" y otras gente de similar ralea, la cultura y la creación morirán irremediablemente. Es decir, nos quedaremos sin libros, sin música, sin películas y sin ningún otro producto de la creatividad humana, que, si hacemos caso a quienes así pronostican el Apocalipsis cultural, se encuentra inevitablemente condicionada a la consecución de algún tipo de recompensa o beneficio, a ser posible pecuniario.

Quienes nos pretenden convencer así no son, en mi opinión, verdaderos creadores, y nunca han sentido la verdadera necesidad de crear que lleva animando al ser humano a hacer avanzar las ciencias y las artes (cosas que vienen en realidad a ser una misma) desde el origen de los tiempos. Admito que un señor que escribe programas de ordenador o graba canciones quiera cobrar por cada copia de su obra que se distribuye. No comparto su opinión pero me parece lícito que así lo haga (aunque también me resulta difícil asimilar el agravio comparativo que supone para otras profesiones que no tienen adquirido ese dudoso derecho a cobrar varias veces un trabajo). Admito también que, ante el previsible declive de su negocio, reivindiquen su condición privilegiada en lo que a remuneración se refiere. Sin embargo, no admito que quieran hacernos creer que de no ser así vamos a perder cultura, porque, sencillamente, no es así. Basta mirar un poco la historia para darse cuenta de que es un argumento falaz. Y sobre todo, y esto es lo que más me irrita, es un argumento que desprestigia y mancha la figura del creador, y yo, como creador que soy, reivindico una imagen distinta de aquellos a los que nos gusta ejercitar nuestra creatividad, ya sea que esta produzca líneas de código, versos, imágenes, bustos de bronce o cualquier otra cosa.

Podría hablar de música o de literatura, por ser formas de creación que me gusta también practicar, pero siendo este un blog de SIG hablaré desde la perspectiva de un tipo de creador que parece que no se tiene demasiado en cuenta: el creador científico. Acabo de hacer pública la primer versión de mi libro libre sobre SIG, un proyecto en el que llevo trabajando ya unos cuantos años, y que es únicamente el fruto de mi afición por los SIG y por escribir acerca de ellos. El libro se distribuye bajo una licencia libre, y mi única intención al escribirlo ha sido disfrutar del hecho de hacerlo. Simplemente eso.

¿Me gustaría ganar mucho dinero vendiendo copias de mi libro de la forma habitual? Probablemente sí (aunque siendo realista es muy poco posible que eso suceda...) ¿Sería justo que yo recibiera algún tipo de compensación de cada persona que saque partido a mi libro, y que pudiera obligar a ello antes de conceder derecho para ese uso? Para algunos, probablemente sí. Para mí, sin embargo, no es ningún problema saber que alguien lo usa sin que yo reciba nada a cambio. En cualquier caso, lo que está claro es que el libro está ahí y ha sido creado sin ayuda de ningún tipo y sin que hayan de mediar otros intereses más allá del que la propia creación en sí comporta. Y, aunque esto suene ciertamente petulante (pido perdón por ello de antemano), creo, como profesional y buen conocedor del SIG que soy, que este libro contribuye a la cultura y al ámbito del SIG tanto como las obras de la mayor parte de esos autoproclamados "creadores" lo hacen a la música, el cine o la literatura. El acto de crear en sí, siempre y cuando no existan obstáculos que lo impidan y se dispongan los medios necesarios (no es lo mismo escribir un libro que hacer una escultura cuyos materiales han de pagarse), es lo que verdaderamente anima al auténtico creador.

Niels Henrik Abel gastó su dinero en publicar su más famoso artículo (el de la imposibilidad de resolver las ecuaciones de grado mayor que el quinto mediante radicales) porque creía en él y porque tal vez su pasión por la matemática era la de un verdadero creador y no la de un artistucho interesado. Lo que Abel necesitaba no eran royalties ni que se le pagara por cada científico que sacase partido a sus trabajo. Más bien, necesitaba que alguien hubiera invertido en darle un trabajo que le evitase tener que vivir en una pobreza que le llevo a morir de tuberculosis con solo 27 años. Es un caso un poco extremo, pero curiosamente esa ayuda a la investigación (es decir, a los creadores científicos) que a Abel le hubiese venido tan bien es la que este mismo gobierno que ahora nos quiere colar la Ley Sinde lleva recortando desde hace tiempo. Cuando toca poner dinero en lugar de recaudar, parece ser que los creadores ya no importan tanto.

Lo que vengo a decir es que la cultura y la creación van a seguir ahí pase lo que pase, lo cual no quiere decir que no se deban apoyar económicamente. Sin embargo, si se quiere de verdad potenciarlas, lo que se debe hacer es apoyar a los creadores para que lleven a cabo su labor creadora, de forma que no tengan que lidiar con otros problemas y puedan dedicarse a su arte plenamente. Y esto se consigue con un apoyo "antes" o "durante" la creación. Darles una forma para vivir del cuento una vez que han creado (es decir, "después" de crear) no es una buena manera de apoyarles y, sobre todo, deja fuera a una buena parte de creadores (entre ellos la inmensa mayoría de creadores científicos) que tienen también mucho que aportar.

Una última recomendación, para aquellos que quieran descubrir lo que la autentica pulsión creadora y científica quiere decir (y que probablemente no les vendría mal a muchos de los pro-Sinde): lean "Surely You're Joking, Mr. Feynman!: Adventures of a Curious Character", de Richard Feynman. Cuentan que Newton se metió una vez una aguja por un ojo para simplemente ver qué pasaba. La curiosidad y la inquietud investigadora de Feynman no tienen nada que envidiar a las de Newton, pero son algo menos espeluznantes y, sin duda, infinitamente más divertidas.

miércoles, enero 05, 2011

JTS 1.11

En el caso de que alguien haya decidido empezar el año probando alguno de los últimos builds diarios de SEXTANTE, y lo haya hecho con gvSIG, habrá notado que SEXTANTE usa ahora la versión 1.11 de JTS, lo cual causa problemas y no arrancará correctamente el programa. Para solucionar esto, deben sustituirse dos ficheros de la carpeta de gvsig ([directorio de gvSIG]/bin/gvSIG/extensiones/com.iver.cit.gvsig/lib:

- bin/jts-1.9gvSIG.jar
- bin/jtsio-1.8.jar

En su lugar, se deben poner los correspondientes a la versión 1.11, que se han de descargar de la página de JTS, ya que no vienen con SEXTANTE (el primero de ellos sí, porque es necesario, pero el segundo no, ya que SEXTANTE no lo usa. No obstante, parece que algunas clases que están ahora en el primero estaban antes en el segundo, lo cual causa conflictos si no se sustituyen ambos).

Una vez hecho esto, no debería haber más problemas. Como comenté en la entrada anterior, JTS 1.11 trae algunas funcionalidades nuevas, que ya están aprovechadas desde los algoritmos de SEXTANTE correspondientes.

Esta información está incluida en el fichero readme.txt que acompaña a la versión diaria.