The format of such custom maps is a series of jpg images with an accompanying Google Earth kml file for the georeferencing information. For Google Earth and for Garmin, the coordinate system must be geographic lat/long, not UTM. That has been the major challenge: producing a seamless array of raster map clippings in correct lat/long pseudo-projection.
It took some time, but is seems to be working now. There is still need for a lot of fine tuning, though.
Raster map export in Google Earth for an area well known to most TransDEM users:
and on a Garmin Montana:

The examples have topo raster maps at 1:50,000 and 1:25,000 plus an aerial image in larger scale. TransDEM optimizes image resolution and image size for efficient use of the limited custom map capacity on the mobile device.