[OAM-talk] OpenAerialMap questions

Sat Dec 8 11:37:10 MST 2007

On Dec 8, 2007 8:13 AM, Christopher Schmidt <crschmidt at crschmidt.net> wrote:
> On Fri, Dec 07, 2007 at 12:09:25PM -0800, Matthew Perry wrote:
> > On Dec 7, 2007 11:15 AM, Christopher Schmidt <crschmidt at crschmidt.net> wrote:
> > The NAIP imagery I am interested in comes as mrsid in UTM projection.
> > (http://new.casil.ucdavis.edu/casil/remote_sensing/naip_2005/county_mosaics/).
> >
> > Is the GDAL version on the server built with Mr Sid support? What
> > about supporting on-the-fly reprojection via mapserver? Or is this
> > just two strikes against this dataset and it will require alot of
> > pre-processing?
>
> Decompressing from MrSID on the fly is slow. Reprojecting on the fly is
> very slow. However, pre-processing data is not something that is some
> kind of permanant black mark against a dataset: indeed, it is my
> expectation that any serious dataset will need lots of reprocessing in
> order to use.

Good points. I was looking for efficiency on the data providers end to
hopefully get more organizations to contribute. But pre-processing is
certainly more efficient on the delivery end.

> The important thing to remember though, is that once it's done once, it
> *never has to be done again* for that dataset. Imagine that the primary
> use case for web mapping is in EPSG:4326 (which is my experience with
> OpenLayers).
>
> > I think supporting other input projections is going to be crucial in
> > the long run since platte carre is almost never the preferred
> > projection for storing imagery for a number of reasons.
>
> I agree with the premise, but not the result.
>

I strongly disagree with that. The Platte Carre projection is
ill-suited for almost every purpose. It is distorted, has inconsistent
scale (no scalebars can be used), can't accurately measure areas, etc.
I'd encourage OAM to at least anticipate the future need to store and
deliver imagery in a more suitable projection.

>
> > > Instead of hosting all the data locally, the longer term plan is that
> > > any sizable dataset for which there is a willing provider will be
> > > populated by connecting to a remote WMS. This 'distributed' style of
> > > data loading is something that we're currently working on building the
> > > infrastructure for, after which there is a plan to document how you
> > > should set up your own data -- after that OSGeo/OpenAerialMap is only
> > > responsible for hosting datasets for which there is no willing provider
> > > available. We have not yet discussed how to scale that aspect of it: my
> > > hope is that we can avoid it for a while, since I think for the most
> > > part there are organizations that are able to provide WMS of most large
> > > datasets, meaning that OAM hosts the small stuff directly, and the big
> > > stuff is shared.
> > >
> >
> > I like the distributed model as well. It's just that I don't
> > personally have the resources to serve up this volume of imagery. But
> > I do know that it's available, how to obtain it, how to work with it
> > and would be willing to set it up if there was a home available for
> > it.
>
> I'll keep that in mind. Is there no university or other organization in
> the area that does have the resources?

Again, I'm thinking of efficiency on the data providers end. Say some
small city has a stock of digital aerial photos in the public domain.
They *could* contribute them but they might conclude that there are
too many roadblocks and expenses in doing so. They are more inclined,
in that case, to keep the data.

But I think you've hit on a good idea; having a network of larger
organizations that could provide the infrastructure and expertise that
the smaller groups need to distribute the imagery via OAM.

> > > > What are the storage limits and scalability of the site?
> > >
> > > I'm not sure what you mean by 'storage limits and scalability'.
>
> Well, it's not entirely clear what it's going to be yet -- but I do know
> that we don't have the > 100TB of disk space it would take to store the
> entire country's NAIP data uncompressed. :) I'm working within the
> constraints of reality, which is important even if it's frustrating :)

Ah that's what I was getting at. So anything under 100 TB is fine? ;-)
... Seriously though, I'm processing the santa barbara county NAIP
imagery right now as a test case. It will be ~35 GB... is this a
reasonable amount of data or would you prefer I find a way to host it
elsewhere?

>
> > > WCS is not
> > > terribly hard, I don't think, just changing a mapfile, though I'm not
> > > sure if it's useful and I'd prefer to have a use case before offering a
> > > service. Downloading data by bounding box is on the 'Things that are
> > > being pondered' plate, but not near term.
> > >
> >
> > Having access to the actual data (not a WMS-ified image) is certainly
> > useful! Let's say I want to create a map of my neighborhood but, for
> > whatever reason, can't rely on a WMS service for that data. Maybe I
> > need to do some manipulation, feature extraction, reprojection,
> > resampling, etc.
>
> Using what toolset? How are you going to get the raw data out via WCS?
> What tool are you going to use to browse the imagery? Does it support
> Geotiffs? Does the output need to be jp2 in order to make senes? does it
> support bigtiff? Etc.

In the past, I've built a web map front end with a little "download"
button which would construct a WCS call based on the current bbox
extent. Pretty simple and it did the trick.. zoom in, hit download,
get a full-res, geo-referenced dataset of that area. As for the
format, I played with a few options and found HFA/imagine format to be
a good fit (for DEMs which we were serving). JP2 was definitely
smaller and more suited to imagery.

>
> Without a concrete use case, I can't tell if WCS is solving an *actual*
> problem or a theoretical problem. Getting the data out in a way that
> anyone who wants to solve an *actual* problem can solve it is clearly
> important: what's not clear yet is whether WCS helps solve actual
> problems in any specific configuration.
>

Again, think of the typical imagery provider. If they upload imagery,
I think it's very reasonable to assume they're going to want access to
that data in the future. And by "access" I mean the ability to
download the actual raw data not a WMS-ified version.

> If you have an actual problem, and can describe the changes that allow
> it to be solved via WCS, then I'll gladly take a suggestion to how to
> make the mapfile support it.

Well if the goal of OAM is to provide access to imagery, I'm not sure
why you wouldn't want to provide access to the data. It doesn't have
to be WCS but some mechanism to download the a geo-referenced dataset
by bbox seems crucial. I showed OAM to many of my colleagues yesterday
and the first question from everyone was "Where do you go to download
an image?".

> And of course, if you don't like that path,
> you can set up your *own* mapserver: the wiki documents that:
>
> http://wiki.openaerialmap.org/Using_With_MapServer
>
> This lets you drop the OAM data into a layer in any mapfile, which you
> can then use to set up your own WCS service, without it needing to live
> on the OAM server.
>
> Am I missing something important here?
>

Well you can't have WCS access to data that is cascaded through WMS.
The whole point of WCS is that it allows access to the actual raw
data. So WCS would only make sense on the first link in the chain (ie
the host server of the data in question).

I'd also like to note that I'm not trying to be difficult here or to
take away from the great accomplishments to date. I'm also not trying
to suggest that these things need to be change immediately. It's just
that, as OAM moves forward, I think it's important to consider the
needs of a broad spectrum of spatial imagery producers and consumers.
I'm just trying to provide a different perspective on these issues
based on my experience.

-- 
Matthew T. Perry
http://www.perrygeo.net