TransDEM has problem with some Canadian DEMs

[Derekc75]

Hi Roland:

I have been working with 1:50000 Canadian DEMs in the Yukon. Several of them work perfectly resolving to 0.75 X 0.75 (Arc seconds) and they stitch together as you can see in this image.



But when I open and convert to UTM the next row of DEMs to the North they resolve to 7.50 X 7.50 (Arc seconds) and when I add the DEM to the previous three I get the following result (the first 3 DEMs are the tiny bunch in the lower left hand corner).



I contacted the Canadian Department that provides the DEMs (NRCAN) and they replied that they had no problem stitching all the DEMs together (they used ArcMap from ESRI and did not convert them to UTM). Here is the image that they provided.



As you can see they successfully stitched all the DEMs together. The good DEMs are 105D02 and the eastern part of 105D03 on the above image. The DEMs that do not work in TransDEM are 105D07 and the eastern part of 105D06 in the above image.

I am providing links for download of the DEMs in question.

105D02 (east and west parts are separate):

http://www.mediafire.com/view/5re7dr6wy ... 0_deme.dem

http://www.mediafire.com/view/tj3pb5bce ... 0_demw.dem

105D03 (east part):

http://www.mediafire.com/view/j232tmplh ... 0_deme.dem

If you open each of these in TransDEM, convert to UTM and save as..., then stitch (Add) them together you will get a perfect DEM identical to the first image. When you open them they resolve to 0.75 X 0.75 as they should. Save these stitched DEMS to a file.

Now download 105D07 (east and west parts):

http://www.mediafire.com/view/ps1t83epk ... 0_deme.dem

http://www.mediafire.com/view/26cfva3hz ... 0_demw.dem

Open them and they resolve to 7.5 X 7.5 which is not correct (at least this happens to me). If you try to add the 105D07 (east part) to the previous three the result looks like the second image. If you try to add the converted 105D07 east and west parts together they will not stitch properly.

Can you tell me what I am doing wrong? I am using the latest 64 bit version of TransDEM.

Thank you,

Derek

[Jerker]

G'day derekc75,

Surely, Derek, the issue is with the first set that is resolving at 0.75 arcsec (and not the second set)? As I understand it, the usual process is to halve the value of the resolution in successive 'improvements'; so one goes from 30 arcsec, down to 15 arcsec, down to 7.5 arcsec! A resolution of 0.75 arcsec would reproduce each grain of dirt on on the ground...

...have you tried to force the incorrect resolution to the proper value?. That is, by manually altering the value in the resolution text box to the one desired, instead of the default value 'suggested' by TransDEM, to ascertain the end result?..

...alternatively, might I suggest that you don't 'split' your workflow. Instead of stopping and saving these DEMs as two parts, continue adding ALL of the individual DEMs until the whole map is created and THEN save the WHOLE map in one fell swoop (if your computer memory will allow this)...

Jerker {:)}

[geophil]

Open them and they resolve to 7.5 X 7.5 which is not correct (at least this happens to me).

Edit:
After the first analysis it looked like a source file issue. In some form it still is, as the 105D07 files use "D" instead instead of "E" as the exponent of floating point data. Needs more investigation.

[Derekc75]

Hi Jerker:

The DEMs that I am using are provided at 1:50000 scale which is already supposed to be 0.75 Arc seconds. The 3.0 Arc second scale in Canada are the 1:250000 DEMs. I only provided a few 1:50000 DEMs as an example. I successfully stitched several more DEMs south of this location to the US border without a problem.

Hi Roland:

Do not get confused with the 105D vs. 105E etc. The entire country of Canada is covered with topographic maps that follow the National Topographic System (NTS) labelling system. The DEMs I am using in this example all occur in the 105D section which is subdivided depending on the scale (1:250000, 1:50000 etc.). At the 1:50000 scale each section (e.g. 105D) is subdivided into 16 subsections, each divided further into an east and west subsection. For instance if you look at the third image above you can see that the next section to the south is labelled 104M etc.

If you follow this link to GeoBase you can see the area of Canada that I am working on with its sections and subsections between the US border in Alaska and Whitehorse in the Yukon:

http://www.geobase.ca/geobase/en/browse ... k&map=105D

I was able to stitch 104M10, 104M11, 104M14, 104M15 to 105D02 and 105D03 without problems!

Jerker: I did what you suggested, not splitting the workflow. I stitched all of 104M10, 104M11, 104M14, 104M15, 105D02, 105D03 together without saving. Here is an image of all those DEMs successfully stitched together:



Then I added one of 105D07 and you can see the drastic results (the DEMs from the previous image get shoved to the lower left corner):





Roland: Could it possibly be a UTM problem? You noted a problem in another thread that I submitted. Check your reply to my first post here:

http://forum.transdem.de/viewtopic.php?f=7&t=285

I appreciate any further assistance.

Thank you,

Derek

[geophil]

Do not get confused with the 105D vs. 105E etc.

No, it's a problem with floating point number representation in the CDED file. These files have a specific text-like format (you can open them in some text editors). All numbers are encoded as text. Certain floating point values use exponential notation. Usually, the exponent is an upper- or lowercase E. That's what many computer languages expect. So does C++, the language TransDEM is written with. The 105D07 files, however, use D instead of E. That's unknown to C++ and the input function ignores the remainder of the text field, hereby screwing up all floating point numbers. 7.5e-1 yields 7.5 instead of 0.75.

I found a hint that Fortran might produce D instead of E when indicating double precision. Not very portable.

Basically I think I have to search for D exponents and substitute them with a proper E before passing the field to the C++ input stream.

[Derekc75]

Do not get confused with the 105D vs. 105E etc.

No, it's a problem with floating point number representation in the CDED file. These files have a specific text-like format (you can open them in some text editors). All numbers are encoded as text. Certain floating point values use exponential notation. Usually, the exponent is an upper- or lowercase E. That's what many computer languages expect. So does C++, the language TransDEM is written with. The 105D07 files, however, use D instead of E. That's unknown to C++ and the input function ignores the remainder of the text field, hereby screwing up all floating point numbers. 7.5e-1 yields 7.5 instead of 0.75.

I found a hint that Fortran might produce D instead of E when indicating double precision. Not very portable.

Basically I think I have to search for D exponents and substitute them with a proper E before passing the field to the C++ input stream.

Roland:

Wow.... OK I am way out of my league here, although I do understand floating point numbers. I appreciate your answering my query. I will wait to until you get back with an answer.

Thank you,


Derek

[Derekc75]

Hi Roland:

My goodness that was easy. I thought about what you wrote. I opened up one of the "good" files in Windows Notepad and it used "e" for the floating point notation. I opened up one of the "bad" DEMs ( 105D07 east) in Notepad and as you wrote there was a "D" for the floating point notation. So I did a "search and replace all" the "D" for "e" and saved the file.

I opened that "good" DEM and added 105D07 east that I had searched and replaced and it works!! They stitch together perfectly. The "bad" DEMs now resolve to 0.75 Arc seconds.

Thank you so much for that hint. I would never have figured that out myself. Now I can search and replace the other "bad" files. I really appreciate your help with this. I will email the Canadian Department NRCAN with an explanation.

Thank you,

Derek

[Derekc75]

Hi Roland:

I followed up with the Canadian Department NRCAN and they confirmed that both D and E have been used:

--------------------------------------------------
Hello Derek

I asked for help from a colleague who worked on CDED project at the beginning.

Unfortunately CDED data sets were made on a long term period with different software. Some used D and some others E.
------------------------------------------------

So now we have the definitive answer. As I mentioned above the simple solution with Canadian DEMs is to use global search and replace of "D" with "e" with a text editor on any files that try to resolve to 7.5 instead of 0.75 (with 1:50000 scale). It is a bother but it solves the issue.

Cheers,

Derek

[geophil]

So it could have been Fortran indeed. I didn't find it mentioned in the specs. Anyway, automatic substitution of "D" exponents with "E" will be implemented in the next update.