TransDEM Forum

TransDEM News, Support, Hints and Resources
It is currently 28 Mar 2024 09:47

All times are UTC + 1 hour




Post new topic Reply to topic  [ 9 posts ] 
Author Message
PostPosted: 15 Sep 2017 05:20 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
I get this error when making a very large TransDEM layout and exporting it to TrainzUTM files, this is the only time I get an error.



I download so many map tiles that the program gets very slow and uses many gigs of ram.

When I open the layout in Tane SP2, the map shows its missing many utm files also...
But the missing files are most of the map that WAS downloaded already in the map tile dialog in TrandDEM.

All the downloaded maps show in TransDEM, but when doing export to UTM, I get an error after TransDEM makes about 1/8 of the UTM files I need for the whole layout. The UTM files that transdem makes, look good, but the missing ones make the rest of the layout un workable.


You can see the result in TANE, most of this video shows the map files that did download, but you can see in the background mountains that they didn't download there.

http://www.youtube.com/watch?v=i_hFEsX8DIc


The ERROR MESSAGE
Image


Top
 Profile  
 
PostPosted: 15 Sep 2017 07:49 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
After finding the search forum doesn't work properly, I made this thread.
I then hunted through pages for answers that might match what im finding, and i found a few.
The search engine for this forum rejects keyword searches, it says they are all common no matter what text entry you use.

Best I have found, is that MY DEM file needs to be 2 times bigger than my selection rectangle, so i redownloaded a DEM file twice the size of my TransDEM selection box, and did a UTM export, this works.




I redownloaded a DEM file from https://gdex.cr.usgs.gov/gdex/ that was MUCH larger than I needed.


MAKE SURE YOUR DEM FILE IS *MUCH* LARGER (3x) THAN YOUR TRAINZ EXPORT AREA SELECTION BOX.

I am not sure if this fixes my problem yet, but it seems it did because once I loaded a larger DEM and ran the UTM export, the progress is making MANY more UTM files than ever before, so im going in the right direction here.

This time the UTM export finished with no error, currently importing into TANE database, will load once finished, and report results here so if anyone else sees this error they can try this.


Top
 Profile  
 
PostPosted: 15 Sep 2017 10:16 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
After that problem was resolved, I found that no matter what selection box I use I get exactly 980 folders and after importing 980 utm objects.

That tells me the program cannot process more than 980 UTM files in one export operation.

So I must select a smaller area, and run multiple UTM export operations?


I'll give that a try next...

The program informs when it has encountered a utm file previously made that it skipped, thats good to know.
Beware, errors look very similar pay attention.


Top
 Profile  
 
PostPosted: 15 Sep 2017 10:57 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
Selecting a smaller area (1/4) the size I was originally using has helped.
I am now showing many more UTM folders than the limit of 980 when trying to use too large a selection area.

It would be good to automate this process a bit more in the programming if its not too complex.
Or a warning to the user that they have selected too large an area for the UTM exporter.
Insanely large routes are possible with todays highend machines, and TANE sp2.




One thing I noticed in the UTM export dialog, is
If you choose The option:
Existing UTM object at same destination : Overwrite Keep KUID

And you have UTM files in that same destination the operation output will say that it skipped the existing ones, not overwrite them.

Shouldn't it overwrite them in case the data in that existing location is older data? Why is it skipping them when it sees they exist, and you have told it to overwrite?


Top
 Profile  
 
PostPosted: 15 Sep 2017 19:36 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1463
From not a very good hotel WiFi:

Yes, you can run the UTM tile exporter several times. It will only create new tiles and skip any existing one. If you do 3D tiles keep in mind that they will consume quite a bit of disk space.


Top
 Profile  
 
PostPosted: 16 Sep 2017 00:39 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
geophil wrote:
From not a very good hotel WiFi:

Yes, you can run the UTM tile exporter several times. It will only create new tiles and skip any existing one. If you do 3D tiles keep in mind that they will consume quite a bit of disk space.



Gosh, my tilemaps are whats killing me, I have used 27gigs of ram with more to go, the disk space is easy to come by.

When trans dem uses that much ram its downloader slows to a crawl, why does it need to keep the map tile in ram after it has already downloaded it?

Using the tile downloader AND having the tiles loaded in trans dem is counter productive, but this is currently done while the tile downloader is doing its download, which is whats causing the tile downloader slowdown when downloading 400+ tiles at 2048 res from google ortho imagery with a api private key.

But it still keeps chugging along even with 27 gigs of ram use.. It does finish the 1,000+ tile download, just takes 2 days.
I am doing this work on a 6 core 4ghz 6850k with 32gigs of ram clocked at 3ghz. When dealing with so much ram, transdem cpu use for its single thread is 100% almost the entire time. It never uses more than 100% of a single cpu, like many programs.

When running the UTM exporter transdem launches multiple copies making that process as efficient as possible, so nice.

So this thread was started because of two problems I was doing.

Problem 1: using a DEM file that was not much bigger than my selection area.
Answer: redownload and use a much larger dem file for your area: https://gdex.cr.usgs.gov/gdex/ I used 1arc data.

Problem 2. selecting too big an export area when doing utm exports.
Answer. the max size limit you should select when doing utm export should be experimented with to determine what works for you, for me its about the size of a mid to large city in America.


Top
 Profile  
 
PostPosted: 16 Sep 2017 22:11 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
From what I understand with the program so far, there is no reason to load that many map tiles.

Due to the fact that the program will not allow more than the 980 UTM tile export, you only need to load up tiles for an area that is a bit bigger than your selection rectangle.

With googles maximum zoom data, which is zoom level 20 or 21 depending on your areas data, the most you need to download is only a few gigs.

I was downloading my entire routes tile maps at zoom level 20. This results in 26 gigs of data in ram. The program is so slow with that much data its painful.



This means the tiles outside your current selection box are just wasting ram and cpu cycles.

Once I understood the two problems I was having, I can now see there is no point in opening that many tiles at once.

The only reason why someone would need a large area to have a map tile(and one that is at zoom level 8 zoomed out far) is so when they export their DEM and ground textures, TANE will show the ground textures in the tane's driver map when zoomed out far.


To recap, loading many zoomed in map tiles for areas LARGER than your current UTM export selection box is rarely needed.
This is not needed because the selection box for exporting UTM tiles is limited.
So one must work in "BLOCKS" in TransDEM.

Choose a specific sized selection box when doing UTM exports and stick to it, and use the coordinate system to keep track of what areas you already exported to UTM file. Write them down so you don't need to try to remember what coordinates you already exported.

Hmm, I see theres a way to make "Rectangular masks."

And its text entry for size and coordinates, so making a grid of rectangular masks that will guide me when doing utm exports will make this much faster and efficient.


Top
 Profile  
 
PostPosted: 17 Sep 2017 11:12 
Offline

Joined: 14 Sep 2017 00:43
Posts: 16
Image
Turns out my DEM file has some data that causes TransDEM to put the track height to invalid levels causing that track in the pic.
Once I ran smooth DEM in transDEM the track was fixed.
That was with 1arcsec NASA data

Which made the following video possible.

A NPR train drives through Johnstown, PA.
https://www.youtube.com/watch?v=wwxVQ7k0Zls


Top
 Profile  
 
PostPosted: 18 Sep 2017 07:21 
Offline

Joined: 05 Jan 2011 16:45
Posts: 1463
From another hotel WiFi:

Even with the best of DEM data, Trainz can only resolve up to 5m horizontally. You will have indications for cuttings and high fills/embankments with hi-res DEMs but your track geometry will still be 2D and track spline point elevation from DEM can never be perfect. With 2D track geometry you will most often end up with roller-coaster track which definitely needs post-processing. TransDEM has built-in functionality for proper 3D track geometry but such data is not normally available to Joe Public.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron

Imprint & Privacy

Powered by phpBB® Forum Software © phpBB Group