There are two aspects here, maximum DEM resolution TransDEM supports and its treatment during export to Trainz.
The nominal maximum horizontal DEM resolution should be 1 m. If your DEM is in Plate Carrée lat/long pseudo projection, the achievable maximum may be somewhat lower. But USGS 1/9 arc sec works well since TransDEM version 2.1. It usually translates to 2 x 3 m. Be aware that the amount of data increases by a factor of 100 if you substitute a 50 m DEM with a 5 m DEM. You will most probably have to split your route projects into smaller modules since TransDEM may run out of memory. The maximum amount of user process memory TransDEM can allocate on 64bit Windows is 4GB. But I agree, details and accuracy of 50 m leaves something to desire. You may have seen
my comparison of USGS NED DEMs. Make sure you get your hi-res DEM files in a format TransDEM can read.
During export to Trainz - as you already said - the maximum resolution will be reduced to 5m, if you select the 5m terrain grid for TS2009 and above or 10 m otherwise. TransDEM always applies a light smoothing filter to the DEM when exporting (low pass), to avoid terracing when using lower resolution DEMs. The effect of this filter should adapt to DEM resolution and should basically be off when working with 5m DEMs. However, I suggest to play around with setting the filter width manually, possibly reducing it to 0 and compare the results.