TransDEM Forum

TransDEM News, Support, Hints and Resources
It is currently 21 Oct 2018 08:13

All times are UTC + 1 hour




Post new topic Reply to topic  [ 9 posts ] 
Author Message
PostPosted: 29 Dec 2017 00:27 
Offline

Joined: 10 Jan 2011 11:31
Posts: 34
Location: Moscow, RU
Good evening.

I've recently got myself some Norwegian DEMs as I decided to try the route building out in TANE. Downloaded both GeoTIFF and .dem formats, which TransDEM doesn't open at all sadly. I was looking forward to taking a look at their 0.25m accuracy tiles. Here are two file sample included, perhaps someone could take a look and help me out? I'm currently on my second day of exporting the combined tiles from Qgis into a .dem and I'm afraid hat will take another week or so. Not certain it would work either :(

https://yadi.sk/d/Kg-0PGHF3R5DHy

The DEM's just give "Cannot open this file" and the GeoTIFFs give this error, i have no idea what it means. Please help.

Quote:
Errors reading file
"D:\Downloads\Norwegian DEM\eksport_76002_20171228\data\dtm\Kristiansand 2014-33-1-434-095-77-dtm.tif"

PCSCitationGeoKey tag not found.
ProjectedCSTypeGeoKey tag value unknown [User-Defined]


Top
 Profile  
 
PostPosted: 01 Jan 2018 16:01 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1103
There are standards and there is interpretation of standards. This makes things complicated.

For the .dem file format, Norway has adopted and adapted that older USGS format. When I first analysed it a few years ago, I found certain meta-data fields set to values not listed in the USGS specs. But I accepted those values and interpreted them as Norway-specific. Now, in your example, at least one of those values has changed, and complies with the USGS specs, but TransDEM did not expect this combination. That can be easily fixed. I hope it's the only change.

GeoTIFF is more complicated. It's not only the PCSCitationGeoKey which is a weird way to specify the spatial coordinate system with a text string. Normally you would use EPSG codes, and UTM/ETRS89 zone 33 is well known in the EPSG coding world (and in TransDEM). Instead they are encoding it in prose, which would require a sophisticated parser unless you associate the entire string with the particular EPSG code, which I do when a user brings such an encoding to my attention. Unfortunately, that's not all. The data representation seems a bit odd, too, but I have not analysed that part yet. The horizontal raster shall be 1m? That's what it tells me and that's what the .dem example says.


Top
 Profile  
 
PostPosted: 04 Jan 2018 08:25 
Offline

Joined: 10 Jan 2011 11:31
Posts: 34
Location: Moscow, RU
geophil wrote:
There are standards and there is interpretation of standards. This makes things complicated.

For the .dem file format, Norway has adopted and adapted that older USGS format. When I first analysed it a few years ago, I found certain meta-data fields set to values not listed in the USGS specs. But I accepted those values and interpreted them as Norway-specific. Now, in your example, at least one of those values has changed, and complies with the USGS specs, but TransDEM did not expect this combination. That can be easily fixed. I hope it's the only change.

GeoTIFF is more complicated. It's not only the PCSCitationGeoKey which is a weird way to specify the spatial coordinate system with a text string. Normally you would use EPSG codes, and UTM/ETRS89 zone 33 is well known in the EPSG coding world (and in TransDEM). Instead they are encoding it in prose, which would require a sophisticated parser unless you associate the entire string with the particular EPSG code, which I do when a user brings such an encoding to my attention. Unfortunately, that's not all. The data representation seems a bit odd, too, but I have not analysed that part yet. The horizontal raster shall be 1m? That's what it tells me and that's what the .dem example says.


Good morning Phil, thanks for your reply and for looking into it. I guess data formats change and progress as we go along. I'm certain I picked 1m spatial resolution for those files yes. They are supposed to be produced from laser data that can go down to 0.25m There's an option to choose such resolution in sets at https://hoydedata.no/LaserInnsyn/ (English interface is available) Would such a high resolution be wise, does TransDEM interpolate it all up to 5 metres of TANE terrain accuracy?

There's also an option to download the whole laser point cloud with insane data sizes and accuracy. I'm a bit lost there to be honest, perhaps you can make heads or tails of the Norwegian system in your spare time? So far I know only enough to select and download a dataset, that interests me. I'd love to see it working in TransDEM as well. If you have any more questions I'd love to work on figuring this out together with you. To the best of my humble abilities.

Cheers.


Top
 Profile  
 
PostPosted: 05 Jan 2018 18:19 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1103
Meanwhile the Norwegian 1m DEMs appear to work in my lab environment with both the "USGS DEM" and the GeoTiff formats. I have to merge that into the release version and prepare a patch. New version will be TransDEM 2.6.2.2, hopefully over the weekend.

Damien_Zhar wrote:
I'm certain I picked 1m spatial resolution for those files yes. They are supposed to be produced from laser data that can go down to 0.25m There's an option to choose such resolution in sets at https://hoydedata.no/LaserInnsyn/ (English interface is available) Would such a high resolution be wise, does TransDEM interpolate it all up to 5 metres of TANE terrain accuracy?
The problem with the hi-res LIDAR DEMs is that they are both great and awful at the same time. They are absolutely fantastic to look at but a pain to deal with because of their gigantic footprint. Downloads are usually hundreds of MB and opening in TransDEM requires lots of physical memory (64bit version is a must).

Trainz can only resolve 5m, that is true, but the other option hoydedata.no offers, DTM10, is lower that Trainz can resolve. So, DTM1 makes sense. TransDEM will automatically reduce 1m DEMs to 5m. You can also do it manually with the respective TransDEM function, but then you would lose some of the visual splendour.

Quote:
There's also an option to download the whole laser point cloud with insane data sizes and accuracy.
That appears to be the raw data, not much use to us. LIDAR raw data processing requires sophisticated algorithms which TransDEM cannot provide.

Quote:
I'm a bit lost there to be honest, perhaps you can make heads or tails of the Norwegian system in your spare time? So far I know only enough to select and download a dataset, that interests me.
What more do you want? :) Anyway, I think I should write a short tutorial and explain the various options as far as we need them.

Here is a section of my test example, city of Bergen, with an OpenTopomap overlay, which you hardly need, as the DEM itself can be read like a map.
Image


Top
 Profile  
 
PostPosted: 05 Jan 2018 19:48 
Offline

Joined: 10 Jan 2011 11:31
Posts: 34
Location: Moscow, RU
geophil wrote:
Meanwhile the Norwegian 1m DEMs appear to work in my lab environment with both the "USGS DEM" and the GeoTiff formats. I have to merge that into the release version and prepare a patch. New version will be TransDEM 2.6.2.2, hopefully over the weekend.


Amazing work Geophil, I really appreciate all the wonderful work you are doing for this community. Can't wait to try out some beautiful, Norwegian landscapes. Just another question, that 1-5 metre upscaling that TransDEM does, does it do so with interpolation, or binning, or maybe in some other way? How do we get five metres accuracy from one?


Top
 Profile  
 
PostPosted: 07 Jan 2018 17:43 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1103
Damien_Zhar wrote:
Just another question, that 1-5 metre upscaling that TransDEM does, does it do so with interpolation, or binning, or maybe in some other way? How do we get five metres accuracy from one?
Well, until now, it was just plain re-sampling, using bilinear interpolation from adjacent source points. While this is correct when reducing DEM grid width, it adds noise when enlarging grid width. With TransDEM 2.6.2.2 it will now be the arithmetic mean value of all source points in range of the target point. This will give a more accurate and smoother result.


Top
 Profile  
 
PostPosted: 11 Jan 2018 13:32 
Offline

Joined: 10 Jan 2011 11:31
Posts: 34
Location: Moscow, RU
geophil wrote:
Damien_Zhar wrote:
Just another question, that 1-5 metre upscaling that TransDEM does, does it do so with interpolation, or binning, or maybe in some other way? How do we get five metres accuracy from one?
Well, until now, it was just plain re-sampling, using bilinear interpolation from adjacent source points. While this is correct when reducing DEM grid width, it adds noise when enlarging grid width. With TransDEM 2.6.2.2 it will now be the arithmetic mean value of all source points in range of the target point. This will give a more accurate and smoother result.


Sounds wonderful, going to try this out ASAP! Perhaps I can finally create that Norwegian route of my dreams? All thanks to you and your marvelous work Geo!


Top
 Profile  
 
PostPosted: 14 Jan 2018 13:07 
Offline

Joined: 10 Jan 2011 11:31
Posts: 34
Location: Moscow, RU
Phil, the DEM's are opening just fine, when we open them as "Open DEM" but when I wish to add additional DEM's from Norway via "Add DEM's" option they give an opening error. So I have to save each file as a Transdem .dem file and then add those together, a minor inconvenience, but still.


Top
 Profile  
 
PostPosted: 14 Jan 2018 15:20 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1103
It's probably the .dem file type. There are tow totally different formats for this. One is MicroDEM .dem, the other USGS .dem.

TransDEM applies the first to all its native DEM files. So, every time you save a DEM as a file in .dem file format, it will be MicroDEM .dem.

The other usage, USGS .dem, was long since abandoned by the USGS itself, but was rediscovered by Canada, and Canadian DEMs are still in this format. They are in geographic coordinates. Years later Norway also discovered it, made a few adjustments to allow UTM projection for Europe. Now, with the 1m data, they added another another tweak. That's why TransDEM couldn't read them. But they still reside in the USGS .dem file format world.

For the Add DEM function in TransDEM to work you always need DEMs in MicroDEM .dem format. The Add function does not have a format converter, as the Open function has. Open DEM is the only function to read all DEM file formats known to TransDEM.

So, you have to open the original Norwegian .dem (USGS .dem) in TransDEM but then save it as new file again. File extension will be .dem once more, but now it will become MicroDEM .dem. Then Add DEM should work.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: Google [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron

Imprint & Privacy

Powered by phpBB® Forum Software © phpBB Group