TransDEM Forum

Since the very first version for Trainz, back in 2005, TransDEM has always supported a “Basemap” feature, generic texture carrier objects, to be filled automatically with images of large scale maps, track schemas or ortho-photography. They are called “UTM tiles” in TransDEM (and adhere to the UMT grid). Since TransDEM 1.2 (2006), these tiles are also placed automatically at the correct position in the route.

Such “Basemaps” are a bypass for the limitations of the Trainz ground textures, which cannot be as detailed as we may want them, not if we paint entire baseboards and the whole route module with map images or ortho-photos.

Like most other “Basemap” implementations, the carrier objects for TransDEM's variant have been prefabricated meshes, which are flat by nature. In hilly terrain, this is less than ideal, because they have to be adjusted to terrain elevation at the point where we currently work.

The textures are individual for each such UTM tile, but the carrier meshes are always the same. Although the original size of 1000 x 1000 m per tile has been complemented with smaller 500 x 500 m tiles in 2010, the smaller ones still have to raised and lowered wherever we are confronted with hill slopes.

It would be much easier for the route builder, if the UTM tiles not only show the high-res cartography or ortho-image but also follow the shape of the terrain. This means, of course, that not only the textures but also the meshes have to be fabricated individually, quite a bit of computation work.

A year or so ago Trainz user ModelerMJ started a project for such terra-formed Basemaps, his approach closely coupled to Google Earth and other Google services. ModelerMJ has now offered to develop a software tool for route builders, based on his earlier work.

Long term TransDEM users may have noticed that I have always been very reluctant to integrate specific Google or other internet service providers' API functionality into TransDEM. The two online services in TransDEM, WMS and Map Tiles, run on on the very basic level of the HTTP protocol, no account registration, no APIs, no cookies, no personal data at all. For the same reason, Google Earth usage is limited to local, offline transfer of simple JPEG images and placemark files.

This policy requires that terra-formed UTM tiles in TransDEM must not rely on Google but have to be created independently of any online service. Of course, the best source for shaping individual UTM tiles is the terrain we already have in our Trainz route.

And that is what I have tried this week. Here is the result, the first tera-formed UTM tile for the well-known Wupper valley at Muengsten:




At the moment, this is only a proof of concept, but I plan to integrate it as a new feature into the next version of TransDEM. It will be called “3D UTM tiles”. Conventional flat tiles will still be offered, then tagged “2D”. 3D and 2D will be compatible as much as possible, including both 1000 and 500 m sizes.

As said in the introduction, 3D tiles are computation intensive. For each tile, three additional steps have to be processed:

1. Retrieve elevations from the Trainz ground file for the position and extent of the individual tile. In TransDEM is easiest way for this is to create small individual DEMs for each. Minimum coding effort here, because the classes already exist.
2. Prepare the individual mesh data. The standard approach for third party programs handling Trainz asset geometry goes via the Trainz-specific clear text XML representation of a mesh. This step computes vertices, normals and texture coordinates in XML format, based on the small DEM. The mesh will have a complete 10 m raster. At the moment I don't think additional optimization to remove unnecessary vertices will be needed.
3. The last step is to generate the actual Trainz mesh – the .im file – from the XML representatione. The importer is provided by N3V/Auran. Users will have to download it themselves, as it may not be redistributed. The idea, of course, is to call the exporter automatically within TransDEM.

If everything goes to plan, the only visual change for the TransDEM user – apart from longer computation – will be a 2D/3D UTM tile radio button in the export dialog.

Two more aspects:

• This spring, I added GeoPDF import functionality to TransDEM which relies on a piece of totally independent external software, the GDAL utilities. It appears that the concept to run GDAL can also be applied to run the Trainz XML Mesh Importer, launched as an individual process, but under supervision of TransDEM as its parent process.
• The three additional steps mentioned are computation-intensive. As I cannot do much about the workflow itself, I have some ideas for parallelization. The last few versions of TransDEM already run some algorithms in parallel and make use of all available CPU cores on the user's PC. The task here will be a bit different as I need more control over the worker threads but I think it's doable.

As always, I can't give any time estimation for it all to be implemented. As indicated above, it's laboratory stage at the moment, proof of concept, with a number of ideas for the final design, but little of the real work has begun.

Roland,

Those results you achieved in your testing are very impressive. The Trainz Mesh importer/exporter is readily available on the Trainz Wiki pages if I'm not mistaken. Blender users who create for Trainz should already have it available too.

One question - Did you encounter any "flashing" of textures between the terrain ground and the 3D UTM tile? Would the UTM tile need to be raised a tiny bit to overcome this (if it occurred)?

Thanks for the look ahead!

Andrew

Yes.
I guess quite a few Trainz route builders will have it already.

Yes, that's the case. I will have to fine-tune the vertical position, adjusting it by a small offset.

This is an amazing breakthrough! Since UTM tiles have always been flat, my entire layout is flat. I raise each UTM tile (carrying a detailed orthophoto) and each track spline 1 meter above the baseboard (to avoid flicker). It looks good, an extensive system of Trainz track laid on flat, photorealistic scenery, but DEM data and 3D UTM tiles will take it to the next level. I can't wait to try it out!

I am still experimenting with a suitable vertical position for the initial placement. 10 cm may already be sufficient, to suppress most flickering while still allowing low objects like trackbeds to poke through the surface.

Proper implementation is now well underway. The texture mesh has become a genuine 3D object, with a corpus and individually triangulated side walls, see the following two screenshots taken with the mesh viewer:





3D tiles will come as either 1000 x 1000 m or groups of 4 with 500 x 500 m, as before with the 2D tiles. (With 500m tiles and a 4096 pixel textures, objects down to 10 cm can be identified.) For the 500 m tiles, anchor points will be the 1000 m UTM grid line intersections, same as before. Apart from the SW tile, the other three in a group of four will be rotated on placement by 90° to 270°. This allows to use the very same textures as for the 2D tiles, not a single line of code needs to be changed in the texture generation part.

I have also added vertex normals for terrain shading in Surveyor, compare this screenshot with those in the previous post.



(The mesh viewer in the first two images appears to be ambient light only, ignoring normals.)

The last screenshot today shows the Muengsten terrain, plastered with 3D tiles. One of them risen above the ground to illustrate the grid of quads. View has been switched to wireframe, with the see-through replacement texture in use.

Outstanding! Is it the nature of the in-game renderer that makes the surface textures look darker than the mesh viewer examples you provided above? Or did you adjust the lighting in Surveyor to emphasize the shadows?

Thanks in advance for another great TransDEM feature!

Andrew

Yes and no.

The shading effect is controlled by my vertex normals, but the intensity is down to the rendering engine and its use of directional lighting. The default scene when starting Surveyor is 10 am, current day and month and World Origin location I think. So, in November in Central Europe the sun is pretty low and shades are long.

I already played around with the normals and reduced their effect to 50% of the geometrically correct value because I thought the result was too intense at first.

Implementation is making further progress and implies a lot of testing. Here are a few more images from the lab:

This is the station area of Menden, a small town in Sauerland, North Rhine Westfalia (NRW). In the foreground we have a 1:1000 track schema from the 1950s, and three other texture sources in the background, further details below.



The 3D tiles are positioned 10cm above ground, high enough to suppress most flicker, but still low enough to make the track spline poke through.


My four texture sources from a vertical perspective with a 100m long ruler to illustrate dimensions:

• Top left: Track schema, 1:1000, manually georeferenced, Prussian cadastre coordinates (Cassini/Soldner projection)
• Bottom left: Bing Maps ortho image (in TransDEM known under its original name Virtual Earth), zoom level 18, ca 1:4000, acquired via the map tile service, automatically georeferenced, spherical Mercator projection.
• Top right: Ortho image, 40cm resolution, NRW state mapping agency, acquired via the agency's WMS, automatically georeferenced, Gauss/Krueger projection. Their images are taken in winter intentionally, longer shades but no foliage, somewhat easier to use for cartographic purposes.
• Bottom right: Topographic map 1:5000 (“Deutsche Grundkarte”), NRW state mapping agency, acquired via the agency's WMS, automatically georeferenced, UTM projection.

I deliberately did not use any Google geo data in this example to emphasize that often other and in many cases better sources exist.

All tiles in this example are of the 500m size. I then pushed the inner four slightly underground to reveal the ground texture. We have the 5m terrain grid here carrying the 1:25,000 topographic map, again acquired from the NRW agency via WMS. At this close distance the ground texture appears blurry.




Zooming out, we get the broader picture, with a now sharper ground texture, and the remaining 3D UTM tiles around, still 10cm above ground.





The variety of sources, map projections and methods to pre-process the data will all be familiar to the TransDEM user. All the math is hidden underneath. And no matter what geo data sources are used, the export to Trainz functions are always the same. Particularly important, generating 3D tiles will not require any additional action by the user – apart from selecting 3D tiles in the first place.



As indicated before, producing 3D UTM tiles with individual meshes requires a lot more computation than simply applying textures to prefabricated 2D tile meshes. To make it run faster I am adding parallelization to the mesh building process. First performance tests are shown in the next two screenshots.

The first is a Process Explorer snapshot showing instances of the Trainz XML Mesh importer as child processes of the TransDEM process. The importer takes about half of the processing time per tile. The maximum number of parallel child processes is defined by the number of cores in the CPU.




The second image shows TransDEM and total CPU load, another Process Explorer sample.



On my machine with 4 cores plus hyper threading all cores are being used well during tile generation. The reason they are not running at 100% load is that the tile generation involves a lot of I/O too, see left graph, and I/O is slowing it down.


Finally the revised look of the Tile Exporter Dialog:



We have a new radio button here: 2D / 3D tiles. If you select 3D tiles two new options will be shown, replacing the 2D tile options. One is the 3D tile height above ground. 10 cm is the default and will already avoid most flicker.

The second option allows to create meshes with a 5m top raster, if any of the baseboard sources for the tile has the 5m terrain grid. Usually this second option will stay off, as it increases mesh complexity and computation time by factor 4. It can be helpful, though, when working with 1/9 arc sec DEMs.

You also need to specify the location of the Trainz XML Mesh Importer. Since the importer does not come with an installer, TransDEM has no easy way to find out where it will have been put. In any case, TransDEM will verify a given location to ensure the importer exists and can be run.

3D UTM tiles will be available with the next release. 3D UTM tiles follow terrain contours and will become much more useful in hilly or mountainous terrain than the flat 2D UTM tiles we have had so far.

As a route building aid, 3D UTM tiles could take over the role which ground textures had until now. That leaves the question how to utilize the ground texture feature in the future. One idea could be to assign a new job to them. How about background textures? If we use a suitable geo data source – ortho imagery springs to mind – we may paint the ground in colours that reflect the landscape of Mother Nature – and save us some work on manual texturing.

For aerial photography the built-in TransDEM ground textures are not adequate. They are optimized for topographic maps which need very few colours. Ortho photos need several more. This requires different textures.

The principle of geo data based ground texturing in Trainz – originally invented by HOG – is to define a palette of monochrome textures and for each baseboard ground vertex select the texture colour from the palette that is closest to the colour of the original (map or ortho image) at this point.

In TransDEM – as in HOG – you can assign your own custom texture palette. ModelerMJ created one such custom texture set in the past, also for ortho images, but I think he used a general purpose collection of colour values (please correct me if I am wrong).

For my ortho imagery texture mapping I went for palette of 128 unique colours (we can use 250 different ground textures per Trainz baseboard. Up to a128 may be taken by TransDEM, leaving at least 122 for your own work). To define the individual colour values, I created a collection of typical ortho photos around the world and processed this with an octree quantizer which will produce my optimised colour set.

This is my ortho image source collection (reduced in size here):




And this is the result after being transformed with the colour palette, to prove that the new palette is working as expected:




The original image has 197352 colours, the result only 128, of course.

And this is the colour palette:



To use a custom texture palette in TransDEM, you will need the ground texture assets and a texture mapping file to assign RGB values to texture KUIDs:

Code:

...

rgb 183,198,180
kuid <KUID2:126773:21208:1>

rgb 226,179,156
kuid <KUID2:126773:21209:1>

rgb 198,185,196
kuid <KUID2:126773:21210:1>

rgb 229,197,155
kuid <KUID2:126773:21211:1>

...

Ortho photos as ground textures will always be quite blurry, since the resolution is only 5 or even 10 m per pixel, i.e. per ground vertex, not really suitable for route building. However, with the 3D UTM tiles we no longer need the ground textures as a surveying aid, instead we can utilize them as a quick way to paint all baseboards with a realistic varnish which – at least for those baseboards in the distance – could be the only ground texturing those baseboards will ever see.

This means, ortho-imagery ground texturing is there to last. These textures may become part of the final route.(There could be copyright issues when redistributing ortho-images, more on this in a future post.) With the longevity of these texture in mind they don't bear the 10m grid and you would also switch off the 1000m grid in a proper route project.

The 3D UTM tiles on the other hand will offer you the 10 m grid as an option.

See the following screen shots:

G'day geophil,

...this is getting better at every post. I am waiting with much anticipation for the next release, Roland. I presume you will be revealing the 'new' texture set at that time, as well and that it will replace the existing one?

Jerker {:)}