[OAM-talk] where to start helping...

[Stephen Woodbridge]

The one thing the comes to my mind when we talk about large datasets, 
large processing, large bandwidth, etc is building something that is 
distributed across a large set of servers and is redundant because 
pieces are on multiple servers.

Think peer-to-peer, SETI at Home, etc. This just seems like a better more 
scalable solution. OAM might then be the central directory, task 
scheduler, whatever.

This way no one company/org needs to bear the heavy lifting. And with 
some advanced planning if someone needs to leave the network that has a 
lot of data, it could be scheduled to be offloaded to other boxes.

Something to think about.

-Steve

Mikel Maron wrote:
> 
> Seems to me that there are a number of distinct problems to hurdle in 
> the OAM domain.
> 
> -There are several very large raster data sets in the public domain 
> (Landsat, USGS aerials) that need processing in a more accesible form.
> -There are myriad smaller raster data sets, like pict'earth/diydrones, 
> and municipal imagery, etc.
> -Mapping APIs want to hook into these data sets in a relatively seemless 
> way through tiles.
> 
> Solving all these problems on one monolithic server, or set of servers, 
> doesn't seem scalable. Unlike OSM with vector data, rasters simply take 
> too much space and too much processing.
> 
> The large datasets are a special case, and there's a public interest in 
> making these widely available. For landsat at least, 
> http://onearth.jpl.nasa.gov/tiled.html is almost there. Perhaps 
> Telascience, or Amazon Public Data Sets, or OSGeo data committee, or 
> some other body could be persuaded to help with hosting. Processing and 
> setup is not a recurring problem.
> 
> The smaller data sets need a more interactive tool set. I haven't looked 
> at the current OAM code, but I take it it's not all the way there. The 
> need puts me in mind of the Map Warper 
> [http://wrp.geothings.net/frontpage], a soon to be open source tool 
> modeled on the MetaCarta rectifier. Warper is forming the basis of the 
> NYPL's historic NYC map project [http://dev.maps.nypl.org/]. This tool 
> handles manual rectification of smallish uploaded images, then produces 
> a catalog and tiles. It doesn't seem like too much of a stretch to build 
> another uploading tool into this system, accepting already georeferenced 
> imagery, and converting it to whatever format Warper uses. This tool can 
> be deployed in a number of different places, to handle different data 
> sets. State of California, diydrones, OSGeo Italia could have their own 
> instances.
> 
> Linking numerous tile sources together is the last problem. Perhaps to 
> start, all of these service just serve tiles for their own areas 
> directly, with OpenLayers and whatever other client handling requesting 
> the proper tile for the area configured for that application. The TMS 
> spec [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification] is 
> designed for advertising tiles in any arbitrary scheme. There's been 
> proposals like distributed tile caching 
> [http://wiki.osgeo.org/wiki/Distributed_Tile_Caching] to integrate 
> distributed sources. I reckon that these ideas are a bit too baroque -- 
> we can pretty much assume clients will want tiles in the standard G-Y-M 
> tiling scheme, i.e. TMS could be simplified. The logic for requesting a 
> tile in a particular location from the appropriate server is pushed to 
> the client.
> 
> I'm just thinking out loud here, haven't looked so much into the 
> technicals or implications. But generally seems to me that something 
> like this approach gives OAM next steps and scalability.
> 
> -Mikel
> 
> 
> 
> 
> 
> 
> *From:* Brent Pedersen <bpederse at gmail.com>
> *To:* talk at openaerialmap.org
> *Sent:* Saturday, January 3, 2009 11:41:04 AM
> *Subject:* [OAM-talk] where to start helping...
> 
> hi, of the hurdles mentioned in the "state of openaerialmap"
> (http://openaerialmap.org/pipermail/talk_openaerialmap.org/2008-December/000055.html),
> i'd like to understand how to help with the programming stuff. chris
> has said this:
> 
> 09:25 < crschmidt> The thing that doesn't exist that should
> 09:26 < crschmidt> is a chunk of code that can take any set of georectified
>                   images
> 09:26 < crschmidt> in any projection
> 09:26 < crschmidt> and make them into a set of 'happy' images
> 09:26 < crschmidt> tiled, overviewed, in the right projection
> 
> so, what's required to do that? and what needs to be determined to start?
> what is the happy projection (4326?)
> what is the happy format (geo-tiff?)
> is it safe to assume it'll be using python-gdal to do this?
> 
> at this point, i dont even know enough to know the proper questions to ask.
> i have messed with the code a bit, and updated to use django > 1.0, but not
> addressed any of the real problems.
> -brentp
> 
> _______________________________________________
> talk mailing list
> talk at openaerialmap.org <mailto:talk at openaerialmap.org>
> http://openaerialmap.org/mailman/listinfo/talk_openaerialmap.org
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> talk mailing list
> talk at openaerialmap.org
> http://openaerialmap.org/mailman/listinfo/talk_openaerialmap.org