Novel Deep Learning Algorithms and Architectures for Hyperspectral Image Processing
Satellite- and airborne-hyperspectral sensors collect images in large number (100-400) of narrow (5-10 nm) and contiguous wavelength bands. Due to their ability to resolve subtle spectral features and unbiased and synoptic coverage of large areas, they have been widely used for composition mapping in a variety of domains covering agriculture, forestry, pedology, geology, planetary studies, hydrology, land use and land cover studies, etc. However, presence of mixed pixels, high correlation among spectral bands, large spectral dimensionality and availability of limited ground truth samples affect proper modelling of spectral characteristics. Moreover, hyperspectral data collected from different platforms, namely, satellite-borne, air-borne and ground-based, have different spectral and spatial resolutions, which need to be fused to optimize information extraction. This research addresses the above stated problems with the specific objective of developing improved machine-learning, in particular, deep-learning based algorithms for (a) spatial-resolution enhancement of hyperspectral images, (b) spectral-resolution enhancement of multispectral images, (c) spectral unmixing and sub-pixel mapping, and (d) spatial-spectral classification of hyperspectral images. Standard benchmark datasets, namely, Indian Pines, Salinas, Kennedy Space Centre, Pavia and Jasper Ridge datasets, and Airborne Visible/Infrared Imaging Spectrometer Next Generation (AVIRIS-NG) dataset collected over Udaipur region of Rajasthan state of India are used to demonstrate the algorithms.
Brief Description of the Research
This research contributes to the extraction of reliable information from the hyperspectral images; particularly it attempts to resolve the issue of the spatial-spectral resolution tradeoff. The focus of this study has been to propose novel approaches, including deep learning architectures and procedures, for addressing the outstanding issues in spectral unmixing, sub-pixel mapping, resolution enhancement and classification of hyperspectral images.
To address the problems related to endmember variability, requirement of pure endmember values, and initialization sensitivity modelling in spectral unmixing of hyperspectral data, a Support Vector machine (SVM)-based approach to estimate the fractional abundances of disparate materials in each pixel’s Instantaneous Field Of View (IFOV) is proposed. A modification of Fuzzy C-Means (FCM) based method for incorporating spectral variability is introduced, which considerably reduces the sensitivity of FCM towards endmember initialization by optimizing the initial seed selection.
Semi-variogram-based and pixel-affinity based approaches are explored for modelling coarse-resolution variations to improve sub-pixel mapping of hyperspectral images. A segmentation-based spectral unmixing algorithm is proposed that addresses the spectral variability and non-convexity of classes; the gradient information is employed to resolve uncertainties in the unmixing process. The proposed algorithms yield accuracy improvements of more than 5% over the state-of-the-art approaches. Further, a deep learning based algorithm is developed that maps the coarse-level pixel affinity representations to finer scale classified maps. A CNN based approach for jointly optimizing the spectral unmixing and sub-pixel mapping stages is also proposed (Results illustrated in Figure 1). Although, the approach yields better results than the conventional approaches, performance depends on the availability of the=e training data.
A 3D-convolution-deconvolution framework is developed to super-resolve hyperspectral images and to reconstruct a set of regularizing features to ensure spectral and spatial fidelity of the data. The proposed loss functions take into account endmembers and spectral priors in addition to the perceptual differences between the images. A collaborative unmixing approach to refine the super-resolution estimate with respect to the input coarse image is also proposed to improve the generalization and learning capability of the proposed super-resolution models (Results illustrated in Figure 2).
A novel framework comprising sparse-coding-based pixel-spectra enhancement, collaborative unmixing, and spatial-spectral prior based transformation is developed for spectral super-resolution of multispectral images to hyperspectral resolutions. Two sparse-coding-based architectures are proposed to project the coarse-resolution pixel-spectra to the target scale. A Convolutional Neural Network (CNN) based encoding-decoding architecture is developed to model the spatial-spectral prior for improving fidelity of the reconstructions. The endmember similarities and spectral image prior are considered while designing the proposed loss functions.
Finally, novel CNN architectures are designed to improve classification of hyperspectral images. A Capsulenet-based framework is developed for modelling spectral and spatial features, including relative locations of spatial features and wavelengths, depth and width of diagnostic spectral features. A convolutional long short-term memory (Conv-LSTM) is employed for sequentially integrating the spatial features learned from each band. The proposed algorithm simultaneously optimizes both feature extraction and classification. The capsule-level integration of spatial and spectral features/patterns yields better accuracy as compared to both ensemble-based and kernel-level integrations. Along with the margin loss, a spectral-angle-based reconstruction loss is also minimized to regularise the learning of network weights.
The developed algorithms can be further optimized for improving their computational efficiency. In the context of super-resolution, a segment-based dictionary learning strategy can be explored where dictionaries and sparse codes will be learned in accordance with the segment. Also, a conditional network strategy, based on gated architecture, can conditionally end the recurrence in the sparse coding sub-network. In both the cases, Capsulenet based iterative routing can be explored to implement conditional connections. The application of the developed super-resolution strategies can be investigated for the interpolation of geophysical and geological data. Recently, a lot of light weight CNN architectures have been proposed. These architectures can be studied for modelling the proposed frameworks towards real time processing. The proposed spectral and spatial super-resolution architectures can be extended towards multi-modal data fusion. Cross-domain constraints can be employed for improving the generalizability of such networks.
Figure 1. Illustration of the sub-pixel mapping on Indian Pines dataset of AVIRIS sensor
Figure 2. Illustration of the super-resolution on Salinas dataset of AVIRIS sensor