Problem Opening USGS 2018 Topo Maps

[Kookaburra]

Normally, the process I follow to utilize a topo map in Trainz starts out like this:

1. Download a GEOPDF from the USGS National Map Download page
2. Open it in Transdem (Open Raster Map)
3. Mask the margins (click "Define Transparent Margins with Polygon Mask" twice)
4. Convert to UTM
5. Save the Georeferenced Raster Map

Works great with topo maps published before 2018. However, when I attempt to open new USGS topo maps (published in 2018), I get unexpected results. In Step 2, Transdem takes a very long time, then asks for information (coordinate system, zone, hemisphere), and in Step 3 it doesn't properly find the margins. Apparently the format of the latest USGS GEOPDFs has changed (they refer to them as GEOSPATIAL PDFs instead of GEOPDFs). Would a newer GDAL package help? I'm still using the GISInternals package I downloaded from the Transdem site.

[geophil]

Apparently the format of the latest USGS GEOPDFs has changed (they refer to them as GEOSPATIAL PDFs instead of GEOPDFs). Would a newer GDAL package help? I'm still using the GISInternals package I downloaded from the Transdem site.

Can you point me to a map sheet that has these issues? I will probably have to analyse this in debug mode. It's possible that they switched to a different coordinate system, one that TransDEM does not recognize. For the margins, or neat-line as they call it, that took me completely by surprise when I discovered this feature in USGS maps. I didn't even know it existed. But it's optional. And I haven't seen it in any GeoPDF map published by other agencies.

In the meantime you can try a newer GDAL build, but I have my doubts. Anyway, that build must include the GeoPDF library.

[Kookaburra]

Thanks. Here's an example, a USGS 7.5 minute map for Felton, CA published in 2018...

https://prd-tnm.s3.amazonaws.com/Staged ... TM_geo.pdf

[Kookaburra]

Just a small update. I've noticed that USGS 2018 "Geospatial PDFs" are perfectly usable in Transdem (and, ultimately, in Trainz) provided that I manually enter the correct coordinate system, zone, and hemisphere, and manually crop the margins. In other words, I can use the new maps. The process just isn't as automated as with older USGS Topo maps.

[geophil]

After the analysis of two of the 2018 edition topo maps I think we are facing three issues with these latest USGS GeoPDFs. Unfortunately, from the TransDEM side we can only handle one.

1. Long processing time: This may have to do with the larger file size but I don't know where that is coming from. Lower compression for the embedded ortho-image? It appears to have the same resolution as older editions (it's still a 1:24k map). The waiting time is caused by gdal_translate. When monitoring its output, we see a lot of error messages

   Code: Select all

   ERROR 1: Pos = 54602588, insufficient arguments for Marked Content
   ERROR 1: Pos = 54602596, insufficient arguments for Marked Content
   ERROR 1: Pos = 54602599, insufficient arguments for Marked Content
   ...
   

   before it is actually starting the output conversion which then takes as long as it did before. I couldn't find any meaningful explanation for this error messages and I don't know whether those errors are relevant in any sense. I have also run the latest GDAL build from the GISInternals site but get the very same results.
2. Geo coordinate system: Here I found a few subtle changes in the meta data description. It's all text and that makes it always difficult to parse. NAD83 has become GCS_North_American_1983, and parameter names which used to be all lower case are now starting with uppercase letters.
3. The neatline (margins):This doesn't make any sense at all now. Some corner points even exceed the overall image dimensions. I cannot imagine that this is by intention. Perhaps we should send a bug report.

One gets the impression that a new software has been employed to produce the 2018 GeoPDFs.

I will prepare a TransDEM update that takes care of the new keywords and spelling in the geo coordinate system encoding. I will also add constraints to the neatline processing so that all given points will lie within the limits of the image. At least you will then be able to manually drag the corner points to the right position.

[Kookaburra]

I decided to re-visit this problem where TransDEM is unable to find and crop out margins in recent USGS GEOPDF topo maps. This time I am using a 2021 USGS GEOPDF. As before, I have installed GDAL 1.11.1 from the TransDEM site (as recommended in the TransDEM documentation). As before, the neatline metadata that GDAL finds in the geopdf doesn't look right, and Transdem can't find the raster margins.

Just for the heck of it, I asked Claude (AI) to find and crop out everything except the actual map image and explain the process. It works. It's interesting to see how Claude tackled the problem.

First of all, here's the prompt I gave to Claude AI:

https://prd-tnm.s3.amazonaws.com/Staged ... TM_geo.pdf
Crop out everything in this geopdf file except the map image, then save it as a new geopdf file. Be sure to preserve all of the georeferencing information in the file. If the neatline metadata in the geopdf file is wrong, fix it. Verify that I will be able to run my GDAL 1.11.1 version of gdalinfo against the new geopdf file without encountering "ERROR 4: Unable to open EPSG support file gcs.csv" (I don't want to have to install support files like gcs.csv). Finally, explain, step-by-step, what you did.

And here's the result, the cropped geopdf:
https://drive.google.com/file/d/1cs1TiD ... drive_link

Here's what Claude AI says it did, step by step. I haven't verified any of its assertions, but they're interesting.

1. Diagnosed the file structure. The original is a 3600×4350 px (150 DPI), 24″×29″ page US Topo GeoPDF with a title/logo header, the actual map inside a neatline border, and a legend/scale-bar footer — plus multiple togglable layers (topo cartography, satellite imagery, shaded relief, PLSS grid), three of which (Images, Shaded Relief, PLSS) are OFF by default.

2. Found the true neatline. GDAL's own NEATLINE metadata tag turned out to be describing the wrong viewport — the PDF actually contains three separate georeferencing viewports (main map, a locator inset, and an adjoining-sheet diagram), and the one GDAL was reading from doesn't correspond to the visible map border at all. I located the real border by detecting the border line pixel-by-pixel (a gray frame at columns 418–3161, rows 360–3773), then confirmed it precisely: converting those pixel corners through the geotransform lands within about a meter of the expected round quad boundary (-121.75°/-121.625°W, 36.875°/37.0°N) — exactly what a 7.5-minute quadrangle should be.

3. Cropped to that pixel window, carrying the same rotation/scale terms in the geotransform (this map is printed true-north-up, which is rotated ~0.79° from the UTM grid — that's normal for USGS quads, not an error).

4. Caught and fixed a rendering trap. My first crop used GDAL's own PDF rasterizer, which ignored the PDF's default layer visibility and silently rendered the satellite-imagery layer instead of the topo cartography. I redid the rasterization with pdftoppm, which respects the PDF's default OCG (layer) state, and rebuilt the crop from that correct image.

5. Rebuilt the GeoPDF using ISO32000 encoding (matching the original's own encoding style) with the corrected NEATLINE polygon.

6. Addressed the GDAL 1.11.1 gcs.csv requirement. I compared byte-for-byte against the original file and found GDAL 3.8.4 had injected an extra /EPSG 26910 shortcut key into the output's /GCS dictionary that the original file doesn't have (the original only stores a full, self-contained WKT string with no EPSG/AUTHORITY references anywhere). That extra key is exactly the kind of thing that sends old GDAL/PROJ4 hunting for gcs.csv. I stripped it with pikepdf, then confirmed the byte string EPSG appears zero times anywhere in the final file — matching the original's proven-safe structure exactly.

7. Verified the result: correct corner coordinates, correct geotransform, correct NEATLINE, and a visual render showing the clean cartographic map with no collar.