TransDEM Forum

TransDEM News, Support, Hints and Resources
It is currently 29 Mar 2024 08:37

All times are UTC + 1 hour




Post new topic Reply to topic  [ 1 post ] 
Author Message
PostPosted: 22 Dec 2013 19:45 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1463
There is another new feature I am currently working on: A 64 bit version of TransDEM.

A year ago, USGS decided to abandon Seamless Server and to offer their excellent DEMs only as prefabricated packages, unfortunately in very large chunks. When at 1/3 arc sec resolution, more or less the standard for US Trainz route projects, you have to plan carefully, because after loading the first of such chunks, the second one may already break the bank. TransDEM offers tools to cut off parts of each chunk, either before (since TransDEM 2.4) or after loading each DEM file, but it sometimes requires more than one attempt to get it right.

Now, with N3V announcing a 64 bit only release for the next version of Trainz and paving the way, I decided to go for 64 bit, too. This will be an additional option and not the replace the 32 bit variant.

The source code will remain the same for both variants and you will benefit from a few improvements primarily intended for 64 bit with the 32 bit version as well.

The 64 bit version will be built for the x64 architecture, the one most common on home PCs (The Itanium architecture, rarely used by consumers, will not be supported.) 64 bit TransDEM requires a 64 bit Windows system (x64), of course. I am testing with Win 7 but it should also run on Vista and possibly even XP 64 (if such systems exist).

My test case is to combine 4 of those 1 x 1 degree chunks with 1/3 arc sec resolution, resampled to a 10 m raster without applying any cropping.

This is the East Broad Top region in the Eastern US, 77 to 79 degrees western longitude, 39 to 41 degrees northern latitude. The whitish rectangle near the centre are four 15 arc min (1:63,000) USGS topo map quads

Image


It turned out that merging became very slow for such gigantic files and I decided to rework several of the DEM processing loops to parallelize them, e.g. merging and UTM zone transformation. It implies that for decent performance you will need at least a quad core CPU to manipulate those DEMs, otherwise large DEM handling won't be fun, not even with 64bit. I also recommend at least 8 GB of main memory, 12 or 16 GB will even be better. This has to do with the TransDEM way of DEM editing that involves a lot of temporary allocation and an undo/redo stack which makes copies of the DEM with every major manipulation step. Without enough memory, some functions will take much longer, as the system has to swap virtual memory to and from disk. What I cannot parallelize is explicit disk I/O, i.e. file reading and writing. Some file formats are serial by nature any way, for others parallel access would actually slow down the I/O. Therefore some patience will still be required.

Even with the 64 bit address space there will remain certain limitations. Maximum DEM size is still about 2 billion (2^31) points. Such a DEM will need 8GB of memory, and, at 5 m resolution (1/9 arc sec) will be good for an area of 150 x 150 km. With lower resolution you will be able to cover an even larger area. I have not tested whether the current Trainz Content Manager can digest a route of such dimensions.

Here are a few observations, taken with Process Explorer:

The TransDEM process proudly presents itself as a 64 bit image:

Image


Process memory after merging the four 1 x 1 degree DEMs:

Image


The original merge loop, a serial matter, merging one file:

Image


After parallelization, merging all three files with the first in the same time. Note the significantly higher CPU usage to make it much quicker:

Image


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 1 post ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 11 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron

Imprint & Privacy

Powered by phpBB® Forum Software © phpBB Group