[OAM-talk] mirroring dataset

[Christopher Schmidt]

On Wed, Jan 09, 2008 at 04:01:51PM +0100, François Van Der Biest wrote:
> Hi list,
> 
> One of our client would be interested in integrating OAM raster data into
> their own data sources.
> I think they'll need to mirror OAM data, since they would like to feed a
> "high load" web server for ponctual events.
> 
> What would be your advices regarding this demand ?
> First, is it possible from an OAM point of view, and, secondly, what would
> be the best way to mirror the dataset (tiles, files ?).

I would say that the right way to do this is to build a cache on the fly
-- probably using TileCache -- using the "Using OAM with MapServer"
instructions. Essentially, this will let them set up a local mapserver
which is based on the OAM cache. (Being based on the cache makaes it
fast.) They can then use a cache locally on their end so that it doesn't
have to overload the OAM server in times when that would be a problem --
though, for the past several weeks (since some performance tuning on the
server) I'm not aware of any significant time where the service could be
overloaded by most web traffic. 

> They need the highest precision dataset for Europe, while the rest of the
> world could just be limited to the "Blue Marble".
> How much data do you think this represents ?

This is somewhat hard for the current infrastructure: If this is the
case, it might make sense for them to treat the OAM datasource as
localized -- possibly using a 'tileindex' in mapserver with a single
item. The single item would be the GDAL_WMS file, and they could then
limit the extents using the rectangular geometry in the tileindex. The
blue marble data is pretty easy to serve -- it's only a couple hundred
meg? less?

I'd suggest that using Memcached as the backend store for their
TileCache would let them set up a couple big-memory servers, cache the
data in that, and it would be expired in an LRU-ish manner. Given that
OAM is typically relatively fast, this would let them be prepared for
'hotspots' due to particular activity, without having to have gigabytes
or terabytes of storage on hand. Even a couple gigabytes of memory would
let them store a relatively wide set of tiles. 

Regards,
-- 
Christopher Schmidt
Web Developer