Scaling FME Engines on WeoGeo

I presented the movie below as part of a presentation at the Safe Software FME User Conference. We had a great time and the Safe crew put on a marvelous show.

The movie shows WeoGeo scaling up to 64 Safe Software distributed FME Engines in the production of tile caches from a world-wide elevation database. The FME Workspace script was created by Dmitri Bagh, and processed on WeoGeo’s FME Constellation built on Amazon Web Services.

The scaling occurred automatically, spinning up FME Engine AMIs, and then shutting them down when the job queue was completed. This is one of our first examples of bringing scalable processing to difficult geospatial tasks.

Examples of the tiles created by Dmitri’s script for Virtual Earth (Bing Maps for Enterprise) and Google Earth can be found here.

Panel 1 (upper left hand corner) refers to the total number of engines in the constellation processing job.

Panel 2 (upper right hand corner) refers to the total constellation utilization percentage. The constellation is polled and when the utilization exceeds the pre-set threshold (50% in this example), it increases (doubles here) the number of engines until it reaches the pre-set maximum number of engines (64 here). The downward spikes occur when each new set of engines are added.

Panel 3 (lower left hand corner) is the average job processing time. There is an increase in velocity when the number of engines exceeds 16, which may be a function of increased overhead costs on the FME Core or bandwidth to the database.

Panel 4 (lower right hand corner) is the total number of jobs completed. 2000 jobs were submitted for this test. The job completion rate accelerates until the maximum number of engines are brought on-line.