You will need some simple vectors in any case. Polylines, marking the course of your railway line, will serve as a baseboard creation filter. Only those baseboards in vicinity of the created will be created, saving hundreds or even thousands of largely waste land boards.
For finding your way around in the DEM terrain you have basically two options: (topo) maps or ortho-imagery. Route builders less familiar with topographic maps go for ortho photos but you will need higher resolution/larger scale than with a topo map. And the way Trainz handles ground textures we face a technical limitation: Maximum usable resolution is 1 pixel per 5 or 10 m. That's good for a topo map 1:25,000 but less so for an aerial image. If you really want to use aerial images, then you better create so-called 3D UTM tiles in TransDEM, which are ordinary Trainz objects serving as high-res texture carriers. TransDEM will create those tiles for you, paint them with your ortho-imagery and also place them into your route. The Muengsten tutorial in the TransDEM Trainz manual shows both options, but uses a large scale 1:5000 map instead of ortho photos.
The tutorial also indicates that you can combine topo maps with ortho-imagery: topo maps as ground textures, ortho images with UTM tiles. You can do it the other way round, too, but in that case the ortho-images are probably intended to become the final ground textures. See here for an example:
viewtopic.php?f=2&t=295TransDEM 2.5 introduced a new ground texture set of its own, 128 unique colours, optimized for aerial photography. I'd prefer that to ModelerMJ's earlier approach which you mentioned.