GIS in XML

Fig 1 – NAIP TX JP2 in ER Viewer 7.2

I ran into a problem at the end of last week. I was attempting to use a set of NAIP county mosaics for Texas. These are very nice 1m GSD images available from the USDA Data Gateway as 4 band JPEG 2000 image files. The 4th band falls into the infrared region and allows analysts to see more vegetation type details. I don’t need it, but it helps out the Dept of Agriculture who is, after all, funding this very useful public imagery. I just wish they published NAIP in an OGC service, WMS or WCS.

Here is a metadata description. These are useful high resolution images and I was providing them for a cable routing contractor to use in his engineering ‘as built’ documents. I’ve done this in the past without much problem using NAIP images from 2006 – 2007 furnished as MrSID compressed files.

The process is to create a corridor in AutoCAD along the proposed route. Next, draw a series of page rectangles along this route making sure the entire corridor is covered, stepping through the route. This is fairly straight forward AutoCAD drawing. After exporting as DXF, these tiles can then be processed through a small Java program to create a batch file for clipping imagery.

This batch file makes use of gdal_translate to clip out a subset of the source image and create a new GeoTiff result. The resulting batch file is just a text file with a lot of lines like this:
gdal_translate -of GTiff -co “TILED=YES” -srcwin 27537 45649 24951 5046.003420858178 C ortho_1-1_1m_j_tx375_2008_2.jp2 potter.tiff

The image files are large so compression is mandatory. Especially since the images are furnished for ftp download from USDA servers. Wavelet compression is very efficient and has become the compression of choice with very large images like these. JP2 is also interesting for web applications due to its progressive transmission by pixel and resolution accuracy. Now that JPEG 2000 implementation libraries have been out for awhile, more and more jp2 files and tools are available. I didn’t anticipate any problem with the 2008 switch from MrSID to JP2.

However, I was very wrong.

A few hours later I had a list of viewers that did not work:

• GDAL: gdal_translate – black except small northern strip
• OpenEV-FW – same
• ER Viewer 7.2 – same
• TatukGIS Viewer – same
• AutoCAD Map 2007 - same
• GeoJasper attempted jp2tif geo.jp2 geo.tif failed
• OPJViewer – fails

and a couple that did: (but didn’t export to tiff gtiff)

• kakadu’s kdu_show.exe
• Global Mapper 11 trial
• ArcView 9.3

Fig2 – NAIP TX JP2 in TatukGIS Viewer	Fig 3 – NAIP TX JP2 in OpenEV Viewer
Fig 4 – NAIP TX JP2 in Kakadu kdu_show	Fig 5 – NAIP TX JP2 in ArcView
Fig 6 – NAIP TX JP2 in Global Mapper

Hmm that is strange. It has a cimicine odor to it, but JPEG2000 has been in the works for years and has a good range of well used tools? NAIP is very popular imagery, however, a quick Google search didn’t turn up any very specific issue. This 11/19/2008 PostMortem ppt lists some JPEG2000 problems but nothing as catastrophic as my problem:

“JPEG2000 Compression format required for 4-band CCM under the NAIP contract. APFO is experiencing the following problems/issues with JPEG 2000 compressions:

• Blurring images.
• Difficulty with program settings
• Limited customer support from open source software libraries
• Image loss with advancing zoom
• Rendering issues due to DVD exceeding ArcGIS 9.1 file capacities
• Rendering issues on other viewers (consumer use) have been resolved.”

This link is an interesting summary from the 11/19/2008 NAIP coordination meeting. Note that the seamless NED resolution has ramifications for processing these higher resolution national imagery programs. Sounds like another reason to move to a national high resolution LiDAR survey of at least the continental US. I suppose LiDAR collection could happen simultaneously with imagery collection and save a bundle of flying costs.

USDA was prompt when I asked about my issue, but since it was viewable in their ArcInfo all was right with the world … and their data. I couldn’t really quibble with that, but I don’t have access to an ArcInfo License so in my world … all is still not right.

Next I took a look at the header records as output from gdalinfo

C:\Program Files\FWTools2.4.2\bin\gdalinfo ortho_1-1_1m_j_tx375_2008_2.jp2
Driver: JP2ECW/ERMapper JPEG2000
Files: ortho_1-1_1m_j_tx375_2008_2.jp2
       ortho_1-1_1m_j_tx375_2008_2.j2w
Size is 59292, 57976
Coordinate System is `'
Origin = (208978.447721050060000,3947556.455350011600000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Corner Coordinates:
Upper Left  (  208978.448, 3947556.455)
Lower Left  (  208978.448, 3889580.455)
Upper Right (  268270.448, 3947556.455)
Lower Right (  268270.448, 3889580.455)
Center      (  238624.448, 3918568.455)
Band 1 Block=59292x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 2 Block=59292x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 3 Block=59292x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 4 Block=59292x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary

There is a small anomaly in the Coordinate System, but all seems fine otherwise. It is interesting to see the Driver listed for gdal is “Driver: JP2ECW/ERMapper JPEG2000″. Erdas ER Mapper makes ER Viewer with the same problem. I don’t know which sdk is licensed by Tatuk and AutoDesk, but possibly they also use the ER Mapper’s JPEG2000 sdk.

Next I fired up Eclipse and took a look at some JAI. First I tried rewriting the jp2 as tif:

package com.mmc.image;

import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;·

public class J2KTest {
public static void main(String[] args) {
try {
      final BufferedImage bi = ImageIO.read(new File("C:/temp/JP2_test.jp2"));
      File dest = new File("C:/temp/test.tif");
      ImageIO.write(bi, "tiff", dest);
      System.out.println("Complete");·
    } catch (IOException e) {
        e.printStackTrace();
    }
  }
}

“no bing!” Just another clue pointing toward the MetaData though:

Exception in thread "main" java.lang.NullPointerException
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KMetadata.replace(J2KMetadata.java:962)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KMetadata.addNode(J2KMetadata.java:631)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KRenderedImageCodecLib.readImageMetadata(J2KRenderedImageCodecLib.java:1006)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KRenderedImageCodecLib.createOriginalSampleModel(J2KRenderedImageCodecLib.java:673)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KRenderedImageCodecLib.(J2KRenderedImageCodecLib.java:261)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KImageReaderCodecLib.read(J2KImageReaderCodecLib.java:364)
	at javax.imageio.ImageIO.read(ImageIO.java:1422)
	at javax.imageio.ImageIO.read(ImageIO.java:1282)
	at com.mmc.image.J2KTest.main(J2KTest.java:19)

Time to take a look at that Coordinate System record in the header using the debugger to look at the IIOMetadata object.

package com.mmc.image;

import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.metadata.IIOMetadata;
import javax.imageio.stream.ImageInputStream;
import com.sun.media.imageioimpl.plugins.jpeg2000.J2KImageReaderSpi;

public class JP2_metadata {
	public static void main(String[] args) {
    try {
	    //final File file = new File("C:/temp/JP2_test.jp2");
			final File file = new File("C:/temp/JP2_test.jp2");

			final ImageInputStream iis = ImageIO.createImageInputStream(file);
			J2KImageReaderSpi readerSPI = new J2KImageReaderSpi();
			final ImageReader reader = readerSPI.createReaderInstance();
			reader.setInput(iis);

			IIOMetadata iioMetadata = reader.getImageMetadata(0);

		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

“no Java!” Just the same problem reading the metadata?

Exception in thread "main" java.lang.NullPointerException
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KMetadata.replace(J2KMetadata.java:962)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KMetadata.addNode(J2KMetadata.java:631)
	at jj2000.j2k.fileformat.reader.FileFormatReader.readFileFormat(FileFormatReader.java:279)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KMetadata.(J2KMetadata.java:135)
	at com.sun.media.imageioimpl.plugins.jpeg2000.J2KImageReader.getImageMetadata(J2KImageReader.java:419)
	at com.mmc.image.JP2_metadata.main(JP2_metadata.java:22)

Too bad I don’t have source code for IIOMetadata. I’d like to step into IIOMetadata and look at header boxes. Perhaps I could actually see where the problem is, maybe even fix it. However, now I’ve put way too much time into this. Also I’m not sure where to find the JP2 sources for JAI ImageIO. Next stop a purchase of Global Mapper and finish the job before I get into trouble. Global Mapper is adequate and it meets my predisposition to Jurassic solutions i.e. cheap.

Summary:

I really tried to find a solution in the Open Source world. Sometimes it just doesn’t work out. Fortunately Global Mapper was able to read the 4band images as a composite and export the images as GTiff. Global Mapper would have taken days for export of full 800Mb jp2 images to GTiff. Fortunately an added bonus let me overlay my clipping page rectangle vectors and then export clipped subsets, which are much smaller and faster. The only drawback is the manual tedium of selecting one clip export at a time. Perhaps, if I knew much about Global Mapper, there is a handy scripting approach. In the meantime, I was able to finish the job on time and I was only a little disappointed about not finding an Open Source solution.

Postscript

USDA added another clue to my puzzle, noting that apparently there is some difference in the interpretation of the standards for JPEG 2000. I imagine that could be the underlying issue, but in the meantime 2008 NAIP 4 band imagery has some anomalous problems with viewer access out here in the wider world.

This entry was posted on Tuesday, August 25th, 2009 at 7:31 am and is filed under GIS, imagery. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.