WeoGeo Table of Contents Tutorial – Part 3
Part 3: Multiple ToC Features mapped to one P5:Customize:Layer
So far in this Table of Contents tutorial, I’ve introduced the ToC file and shown how to create a WeoGeo Listing that uses it. The Dataset that was created in Part 2 had two P5:Customize:Layers with 1:1 corresponding ToC Features. This time, we’ll create a Dataset that has many ToC Features mapped to one P5:Customize:Layer.
Overview
Recall that one of the Properties of the ToC’s Features was “LAYERS”.
The LAYERS values map to the entries in the P5:Customize:Layer. Each P5:Customize:Layer can have one to many ToC Features. In the example below, the Costa Rica and El Salvador Features both have LAYER values of 0: many LAYERS map to the same P5:Customize:Layer. If the P5:Customize:Layer with index zero is selected, any others with LAYERS: 0 are selected for processing. (Note however that some files could still be excluded from delivery if the User selects Spatial Select/Clipping and the P5:BBOX that the User selects doesn’t intersect the Feature.)
An Example
Let’s look at an example. This WeoGeo Market Listing is looks similar to the one used in ToC Tutorial Part 2, but the Natural Earth states_provinces have been split into separate files for each country in Central America.
As we saw in the screen-shot above, There is a corresponding ToC Feature for each states_provinces SHP file and the LAYER value for that Feature is always “0″. This maps to the 0 in the 5:Customize:Layer, which is defined in the WeoFile’s <layers> tag (it is space delimited):
<layers>States_Provinces Countries</layers>
Note that the Natural Earth ca_10m_admin_0_countries.shp is also available in position 1 with <layers> value of “Countries”.
Script Modifications
Now let’s make a few changes to the Python script that we used in Part 2 to create this type of ToC. Instead of passing a list of files to the “toc_example” function, we pass a dictionary with the file name as Key and the Layer value (e.g. 0 or 1 in this example) as the Value.
Lines 123-139 perform that file searching and dictionary construction:
The dictonary looks like this when pretty printed:
{‘./ToC_tutorial_part3\\Belize_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\Costa_Rica_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\El_Salvador_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\Guatemala_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\Honduras_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\Nicaragua_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\Panama_ca_10m_admin_1_states_provinces.shp’: 0,
‘./ToC_tutorial_part3\\ca_10m_admin_0_countries.shp’: 1}
Inside the toc_example function on lines 46-50, the dictionary keys are extracted to the same “files” list variable that was used in Part 2 (to minimize code changes) and the associated values are extracted a new “layers” variable using list comprehension:
Finally, on lines 82-99, the new layers list variable is used:
You can view and download the full source for this from WeoGeo’s Public Code repository.
That wraps up the many-files : one ToC Feature example. In the next installment of this tutorial, we’ll explore more advanced features of the ToC.
Portland, OR – July 27, 2010
WeoGeo, a leader in geospatial content management and Software-as-a-Service (SaaS) solutions for the geospatial data industry, announces today that Pitney Bowes Business Insights (PBBI) has selected WeoGeo’s Platform to build its Geosk™ Marketplace and Geosk™ Library services.
The PBBI Geosk Marketplace provides access to a comprehensive data catalog that includes both free and fee-based geospatial data from Pitney Bowes Business Insight as well as content from third-party data providers such as the Ordnance Survey and TomTom. For data providers, Geosk Marketplace will function as an online delivery channel for their products, providing unprecedented access to data consumers around the globe. The new content management service, Geosk Library, will also enable enterprise organizations to store, share and manage geospatial data from a cloud-based content library. This library can also be available from an appliance for organizations with strict security policies outside the firewall.
“WeoGeo is an excellent fit as a Strategic Technology Partner for PBBI,” said Scott Robinson, Director, Global Data Products at Pitney Bowes Business Insight. “Its innovative and intuitive products integrate Software as a Service, cloud computing, GIS/mapping, and spatial data vending into a seamless offering. These products, and their integration, are critical to us as we build our on-demand data products and content management portfolio offerings.”
“The ability to effortlessly find and deliver geo-content and services to employees, customers, or stakeholders is our value proposition,” said Paul Bissett, CEO and Co-Founder of WeoGeo. “PBBI saw that WeoGeo’s market positioning and product suite could power their vision for a geospatial marketplace and content management services. We are excited with the opportunity to help bring PBBI’s Geosk Platform to their customers as well as acquiring new ones.”
For Further Information, please contact:
James Fee
WeoGeo, Inc.
480-225-2287
jfee [at] weogeo [dot] com
About WeoGeo, Inc.
WeoGeo is the leading provider of on-demand content management and marketplace services for the spatial data industry. Anyone who considers “place” a part of their value-added information stream can now rapidly store, search, transform, and sell their professional mapping, demographic, GIS, CAD, or other spatially aware digital content. WeoGeo provides scalable storage, indexing, and hosting solutions for an organization’s valuable digital files under a flexible payment plan that dramatically reduces IT infrastructure costs. Their Software-as-a-Service solutions expand the reach and productivity of small and large enterprises as they seek to share and monetize their digital geo-assets. For more information visit: http://www.WeoGeo.com
About Pitney Bowes Business Insight
Pitney Bowes Business Insight, a division of Pitney Bowes Software Inc., a wholly-owned subsidiary of Pitney Bowes Inc., helps organizations to acquire, serve and grow customer relationships. Our leading-edge solutions in the areas of Customer Intelligence, Customer Communications and Customer Care enhance our customers’ operational systems and workflows, enabling them to manage their customer relationships more effectively. We offer unique and compelling capabilities, including location intelligence; GIS; predictive analytics; data quality, management and integration; and customer communications management. In combination, our solutions and capabilities deliver customer insights that create competitive advantages. Leading companies, government agencies and systems integrators rely on our global expertise and decades of leadership to improve their operational effectiveness and business results. Visit http://www.pbinsight.com and http://www.pb.com for more information.
Pitney Bowes Business Insight’s press release is here.
Faster Tag Searching in WeoGeo
I find the WeoGeo method of finding data using maps much quicker than textual based searches. Using a map to define boundaries on data is much more accurate and a ton quicker than a similar textual based approach. That said, you still need to narrow down your search criteria with tags to zero into specific data you want.
With our latest release, searching with tags returns results instantaneously. When you are browsing results of a WeoGeo search, click on the Advanced Filter tab at the bottom and type in any tags. Quickly you can narrow down your search and get downloading.
WeoGeo discovery tools can keep up with your workflows.
If there is one thing that I have learned over the years, it is that nothing is easy. Sure sure, they may look easy on the surface, but deep down, they can get pretty hairy. , I am referring to the world that I eat, sleep and breath, …the geospatial world of course. You all know just as well as I do that simple lines on a map lead to complex databases and business processes that make the hair on the back of your neck stand straight up. So, how do we share data if it is so complex? I’m willing to bet that most of you out there have used FTP to share data. I took the liberty of outlining why I think this causes problems for effective workflows here. Also, if you think about it, throwing up one link to a zipped dataset (even a multi-file one) is easy, but you can’t customize it, you can’t deliver the whole thing (possibly GBs of data), much less break them out into a bunch of individual datasets. Well, I am excited to say, life in the geospatial world, just got a whole lot easier. This very issue is something we’ve been attacking at WeoGeo. We asked ourselves, “How can you share multi-file datasets, but still allow customization of what is delivered”?
Maps usually look like this. Multiple layers make up a product.
The answer? By creating a new feature, called the Table of Contents (or ToC). Our new ToC is part of our latest release coming out this week. You can also check out Dan Dye’s blog about WeoGeo support for the Table of Contents that was posted earlier this month. The ToC is what we use to allow users to upload multiple file datasets to WeoGeo. While we’ve seen web services that have multiple datasets in them, there hasn’t been a solution to date that gives users the ability to customize the data to suit their own needs/wants and then deliver that customized file back to the user. The uniqueness of WeoGeo’s solution helps deliver these datasets to end-users without the need for the dataset provider to do anything beyond listing or uploading their data to WeoGeo.
Once you upload your dataset with the ToC, users can customize the product by selecting only the layers they wish to download.
Let’s go through how this works. So, like you, most maps I create have multiple layers in them. For example, I might have some simple background data from Natural Earth, some road data from TIGER and some environmental data from the US Fish and Wildlife Service. These multiple datasets blended together make my map. I could of course upload all these datasets individually, but that could mean tens of layers that a user would have to download to replicate my map. With the WeoGeo ToC, I can put all those datasets into one WeoGeo entry, including cartography (MXD, LYR, Mapnik) allowing users to download the data all at once and perform the customization themselves, picking and choosing only the layers they actually want in their end product.
The "map sandwich" can reside on your WeoGeo Library
Photo by roboppy – http://flic.kr/p/eDy3A
Once you start using the ToC feature, you won’t ever want to hear the words zip file, or FTP again. I know first hand that trying to break up maps into discrete datasets is sheer lunacy. What you need is the ability to share exactly what is on your GIS desktop with whomever needs or wants it. The ToC gives WeoGeo users the ability to do this, and much much more. Replicating your GIS projects into the WeoGeo Library content management system is as easy as pie with ToC. You can see this ToC in action with this Natural Earth dataset or learn how to create one yourself on our Support Wiki.
Give me some of that tasty WeoGeo Pie!
Photo by sea turtle – http://flic.kr/p/5zZ3aC
Since our May 2010 release, we have been working under the hood to improve our search functionality and add a variety of new features. The latest release on July 19, 2010, brings all of these improvements and features to life. Please refer to our release notes for details.
Highlights:
1. Faster Advanced Filter functionality and more efficient search capability. Previously, filtering by tags was time consuming, however with our new improvements, searching with tags returns results instantaneously.
2. Support for XYZ tiles: Providers can now upload XYZ tilesets instead of kamap tiles.
3. Support for Spherical mercator as a preview projection value.
4. Table of Contents improvements. Check out the Table of Contents wiki page to learn more.
Please note that if you are a developer actively working with our APIs, a first draft of the XSD schema for the dataset weofile returned with the Get Dataset as WeoFile API call is available here.
Do you know we have video tutorials on our wiki? You can also check WeoGeo’s YouTube channel to access them all in one place.
WeoGeo Table of Contents Tutorial – Part 2
Part 2: Multiple Vector Files in a single WeoGeo Dataset Listing
Now that we have covered the purpose and workings of the ToC, let’s take a look at how to produce one and create a WeoGeo Listing that uses it. You can obtain everything that is required to follow along in this tutorial by ordering this dataset on WeoGeo Market.
First, let’s take a look at the Python script that is included in that data and is used to generate the ToC. This script can be invoked as a stand-alone module or by importing into another script. In this tutorial, we’ll hard-code the locations of the data files and call it as a stand alone program
Below line 114, update the files list and out_file_name so that they contain the complete path to the Shapefiles used in this tutorial and the location where you want the ToC to be written.
Now execute the script with:
python toc_example.py
Note: If you get an error when attempting to “import ogr”, then you’ll need to install it (how have you lived without it?). You can find instructions for installing GDAL at the Python Package Index.
Details on toc_example.py
The rest of this tutorial will focus on the nitty gritty details of how the toc_example.py works its magic. Feel free to stop here if you aren’t a WeoGeo Developer or Pythonista.
The first thing that the toc_example function does is to create empty dictionary.
All of the JSON elements (Features, etc.) will be added to that dictionary. At the end the dictionary is encoded into JSON and then written to a file.
Simple right? In between those two code blocks, we create a Feature for each Vector file and finally create the ToC Feature that is used as a look-up.
For the features, we start with an empty list: features = []
There is a for loop to perform the following actions on each vector file (also see lines 48-82 in the complete source below):
1) get a list of its “amigos” (e.g. for a shapefile, [“shp”,”dbf”,”shx”,”prj”]
2) get its Minimum Bounding Rectangle (MBR) (this is why ogr is required)
3) append the above (and some other hard-coded stuff that we will neglect for now) to the features list.
Creating a Listing that uses the ToC
Now that you’ve produced the ToC file, it needs to be attached to the WeoGeo Listing as a data_file. This is accomplished by appending the ToC file to the datasets/data_files list in the WeoFile with the name WeoGeoTableOfContents.json (note that it must have exactly this name).
One other change is required to the WeoFile that was included in the sample data before you can use it: change the dataset/weogeo_host tag to point to your WeoGeo Library.
Lastly, use the following flags when invoking the WeoApp to upload the dataset to your library:
[bash]
weoapp –no-xml-validation –no-delete –no-kml –no-interpolated-previews –no-preview –no-tiles –no-info weofile.weo
[/bash]
These flags tell the WeoApp not to produce Preview Images from the vector data. You can either make your own or use the “kamap.zip” tileset that I provided in the tutorial data.
That is it for part two! In the next Table of Contents tutorial, we will use the ToC for Attribute instead of File/Layer selection.
Here is the complete code for toc_example.py Update: You can view and download the full source for this using WeoGeo’s Public Code repository.
WeoGeo Table of Contents Tutorial – Part 1
Part 1: Table of Contents Overview
UPDATE: Also view parts 2 and 3 of this tutorial:
- Part 2: Multiple Vector Files in a single WeoGeo Dataset Listing
- Part 3: Multiple ToC Features mapped to one P5:Customize:Layer
Introduction
When WeoGeo initially introduced support for customization of vector listings, there was a limitation: only a single vector file in each Dataset. With the recent introduction of Table of Contents (ToC) files, that limitation has been removed. (As always, we release early and release often.) In this tutorial, I will introduce the ToC and show how it can be used to provide Users of WeoGeo Library (or Market) options for selecting from one or more files. Subsequent tutorials will show more advanced features such as attribute selection and how to map a single Panel5:Customize:Layer to multiple vector files.
First and foremost, I want to be clear that the ToC is *not* used by the WeoGeo web client or any other clients (e.g. the Toolbar for ArcGIS). The ToC is never transmitted to these clients and WeoGeo Users never see the ToC. The ToC is exclusively server-side and is only used by the FME Workspaces that perform the requested customization (e.g. layer selection, attribute selection, clipping, etc.). WeoGeo Users make their selection (of Layers, Attributes, etc.) using the Panel5:Customize:Layers button.
P5:Customize:Layers is populated with the values that the Library Admin writes into the datasets/layers XML tags in the WeoFile and can later be edited using the Library Admin Console or with the Dataset Update API.
ToC Basic Concepts
Now that we’ve obviated misconceptions about the consumption of the ToC in the client, let’s talk about the ToC itself. The ToC is a GeoJSON file. The ToC performs a role similar to an XML manifest file, but because it is GeoJSON, it is also a valid spatial dataset itself. There is one GeoJSON Feature for each vector file that the Listing contains. Each feature has a polygon geometry, with the Minimum Bounding Rectangle of the vector (thus the polygon) that the Feature represents. In the feature’s properties, a LAYERS parameter contains an index, which is used to match the Layers that the User selects.
For example, the tutorial data has two features: 1) countries and 2) states/provinces. The Layer Parameter Values in the ToC are 0 and 1.
In later tutorials, we will see that one P5:Layer can actually map to multiple ToC Features but for now, let’s stick to this example where one P5:Customize:Layer maps to one ToC:Feature.
ToC and Pricing
When the User selects one layer or the other, only that layer is delivered. How does this affect pricing of datasets that are listed on WeoGeo Market? Currently, each layer is treated as an equal contributor to the total price that the Provider has specified. For example, if I have priced my dataset at $100, and the User selects only one of two layers (and clipping is turned off), then the price will be $50 (plus Transaction Fees).
That concludes the technical introduction to the WeoGeo Table of Contents. In the next part of this tutorial, we’ll produce a Table of Contents file with a Python script and create a WeoGeo Listing that uses it.
WeoGeo @ 2010 ESRI International User Conference
It is about to get very cloudy in San Diego, at least the good kind of cloudy.
Photo by The Brit_2 - http://flic.kr/p/7HWcwQ
WeoGeo will be exhibiting at the 2010 ESRI International User Conference in San Diego, CA. Swing by Booth 2716 on the north end of the exhibit hall near the Federal Showcase. We’ll be showing off our new WeoGeo Tools for ArcGIS toolbar along with our brand new release features and API’s. Or just come by and see Paul, Dan and James anytime the exhibit hall is open.
Make sure you follow @weogeo on Twitter or Like us on Facebook to stay on top of all our announcements during the week.
ArcPy background processing in ArcGIS v10
I’m exploring the new ArcPy site package in ArcGIS 9.4 10.0 and I really like the background processing. For example, to find a reasonable level of generalization for a complex shapefile, I’ve previously either used the hit-or-miss approach (which is tedious and time consuming) or batch produced many combinations and then picked from the results. However, the visualization of the numerous products in the second approach can also be tedious. However, with background processing capability of ArcPy, I can kick off the batch processing and explore the results as they are produced. Check out the code-snip and video below.
[python]
for tol_cnt in range(10):
tolerance = tol_cnt * 4000 + 5000
target = r"S:\bend_simplify_%s_k_ft" % (tolerance/1000)
CA.SimplifyPolygon(aus, target, "BEND_SIMPLIFY", "%s Feet" % tolerance, "", "RESOLVE_ERRORS", "NO_KEEP")
[/python]
[swf src="http://images.dandye.blogs.weogeo.com/ArcPy_background_processing_in_ArcGIS_v10.swf" width=407 height=360]
WeoGeo is Out of Private Beta
WeoGeo, Inc. is excited to announce that with the latest release on May 6th, 2010, we are officially out of private beta. Invite codes are no longer needed and we are moving full speed ahead. To see just how far we have come, check out James Fee’s latest blog post, Getting Started With WeoGeo. Better yet, see first hand and sign up here and try us for free for the first 30 days.
An important note to our current account holders, WeoGeo is grateful for all of the support and testing that you have contributed during our private beta period. It is your feedback that has enabled us to reach our goal. Thanks to you, we can proudly release our SaaS Library to the public. Also, please mark your calendars, the 30-day clock will begin on June 1, 2010 with charges to begin on July 1, 2010.
Note-worthy features found in this release:
- An independent developer environment at www.weogeo.net. More information is available here.
- WeoSuite (WeoApp/gWeoApp) reaches version 1.0.0.
- Support for Custom Licenses is finally here! This feature was the most frequently requested item by our existing customers. With this support, you can set the license terms independent of WeoGeo’s standard license.
- Allow providers to include multiple related files in a single vector upload via Table of Contents (ToC). For more details, please check our wiki page.
All details pertaining to this release can be found here. To begin using these new features, make sure to download the latest version of WeoApp here: http://downloads.weogeo.com/index.html.
Also, please stay tuned for more exciting features on the way. We’ve got much more planned in the next few weeks and months.
We would love to hear about any questions, comments, feature requests, bug reports etc. you have. Please contact us here. Attending PBBI Insights 2010 User Conference? Come say hi to Paul Bissett and James Fee.
Subscribe via RSS
Subscribe via email
