Multi-Scale Dense Graph Attention Network for Hyperspectral Classification

Abstract

In recent years, numerous deep learning-based methods have gained increasing attention in hyperspectral classification, particularly the Graph Neural Network, which exhibits superior capabilities in structural description. However, a single graph structure is not suitable for hyperspectral feature representation. Therefore, we propose a novel Multiple-Scale graph network structure, known as the Multi-Scale Dense Graph Attention network for hyperspectral classification. Firstly, semi-supervised local Fisher discriminant analysis and superpixel segmentation were employed for dimensionality reduction and multi-scale graph construction, respectively. Secondly, Spectral-Spatial convolution is applied to extract shallow features from the image. Subsequently, an improved graph self-attention network is sequentially applied to each scale graph, and the different scale graphs are densely connected through spatial feature alignment modules, designed using twice matrix multiplication. Finally, the combined pixel-level feature map from multiple graph spaces is derived, and Spectral-Spatial convolution is employed to fuse the abundant feature maps for hyperspectral classification. Experimental results on various hyperspectral datasets demonstrate the superiority of our MSDesGATnet over many state-of-the-art methods. The code is available at https://github.com/l7170/MSDesGAT.git.

RÉSUMÉ

Ces dernières années, de nombreuses méthodes basées sur l’apprentissage profond ont suscité une attention croissante dans la classification hyperspectrale, en particulier le réseau neuronal graphique, qui présente des capacités supérieures en termes de description structurelle. Cependant, une seule structure de graphe n’est pas adaptée à la représentation des caractéristiques hyperspectrales. C’est pourquoi nous proposons une nouvelle structure de réseau graphique à plusieurs échelles, connue sous le nom de réseau d’attention graphique dense à plusieurs échelles pour la classification hyperspectrale. Tout d’abord, une analyze discriminante de Fisher locale semi-supervisée et une segmentation de superpixel ont été utilisées respectivement pour la réduction de la dimensionnalité et la construction de graphes à plusieurs échelles. Ensuite, une convolution spectrale-spatiale est appliquée pour extraire des caractéristiques de niveau supérieur à partir de l’image. Par la suite, un réseau d’attention graphique amélioré est appliqué séquentiellement à chaque graphe à différentes échelles, et les graphes à différentes échelles sont connectés de manière dense grâce à des modules d’alignement des caractéristiques spatiales, conçus à l’aide de deux multiplications matricielles. Enfin, la carte des caractéristiques combinées au niveau des pixels à partir de plusieurs espaces de graphe est obtenue, et une convolution spectrale-spatiale est utilisée pour fusionner les nombreuses cartes de caractéristiques pour la classification hyperspectrale. Les résultats expérimentaux sur divers ensembles de données hyperspectrales démontrent la supériorité de notre réseau MSDesGAT par rapport à de nombreuses méthodes de pointe. Le code est disponible sur https://github.com/l7170/MSDesGAT.git.

Introduction

Hyperspectral imagery (HSI) combines spatial information with hundreds of continuous, narrow spectral bands, enabling the extraction of more discriminative features for precise land cover classification. However, three main limitations persist in HSI classification: (1) Redundancy in spectral features within hyperspectral data. (2) Limited availability of training samples. (3) Significant spatial variability in spectral signatures (Camps-Valls and Bruzzone 2005). To reduce the impact of high-dimensional redundancy, the focus primarily lies on obtaining a low-dimensional representation of high-dimensional data samples while retaining most of the intrinsic information. Techniques such as Principal Component Analysis, Fisher Discriminant Analysis, Local FDA (Li et al. 2012), manifold learning, Graph representation (Li et al. 2016), and Semi-supervised Local Fisher Discriminant Analysis (Li et al. 2017) have been employed. In the case of HSI classification with limited sample size, methods such as sample augmentation (Li et al. 2018), semi-supervised learning (Aydemir and Bilgin 2019), adversarial network (Huang et al. 2022), few-shot learning (Liu et al. 2022), superpixel segmentation-based approaches (Zheng et al. 2019), Pseudo-labeling combined with pre-trained model (Yao, Hong, et al. 2023) and enhanced prototype contrastive learning (Liu et al. 2023) are utilized. To address the third limitation, Support Vector Machine (Mercier and Lennon 2003) and Random Forest algorithms (Amini et al. 2014), along with incorporating spatial information such as Markov random field (Tarabalka et al. 2010), Random walk (Cao and Wang 2017), and guided filtering (Kang et al. 2014; Liao et al. 2018), have achieved significant success in traditional machine learning-based HSI classification over the past decade.

In recent years, deep learning methods have demonstrated promising performance in HSI classification. Notably, Convolutional Neural Networks (CNNs) such as 1D-CNN, 2D-CNN, 3D-CNN (Audebert et al. 2019), or Hybrid-CNN (Roy et al. 2020; Datta et al. 2022) have been successful in effectively extracting spatial and spectral features at the pixel level. These features are then utilized in fully connected neural networks for classification. At the same time, multimodal-based methods have become the trend, such as the use of multimodal data (Yao, Hong, et al. 2023) while using advanced supervised classification methods such as transformer (Zhang et al. 2023; Yao, Zhang, et al. 2023) or unsupervised methods (UCSL; Yao, Zhang, et al. 2023), have achieved significant results. However, CNNs with fixed-size convolution kernels can only process regular grid data, making it challenging to fit different sizes of ground objects in HSIs. To address this, some researchers proposed Multiscale Feature Fusion Networks based on global weighting (MSFGW; Wang et al. 2023) and Non-Gridding Multi-level Concatenated Atrous Pyramid Convolution (NG-APC; Zhang et al. 2019). These methods employ atrous convolution kernels to process multiscale objects. While different convolution kernels can be combined (MJ-DNN; Liu et al. 2022), the features extracted exhibit spatial scale invariance. Consequently, some scholars have suggested HSI classification models based on Graph Neural Networks (GNNs). GNNs offer the flexibility to organize nodes that describe the relationships among ground objects in the spatial or spectral structure. For instance, Spectral-Spatial Graph Convolutional Networks (SSGCN; Qin et al. 2019) treat each super-pixel in each spectral band as a graph node, albeit resulting in redundant computations. Among GNN frameworks, Graph Convolutional Neural Network (GCN) can aggregate the spectral features of ground objects and effectively improve the spatial variability of HSI spectral signatures (Hong et al. 2021; Hu et al. 2022). However, GCN relies on graph structure and is not applicable to HSI scenes with ground objects of different sizes, which need to be described by multi-scale graph structure, and at the same time, with the increasing number of GCN layers, the oversmoothing problem occurs, which has been alleviated by some advanced GCN-based methods using DropEdge technique and residual connection (Yu et al. 2023) in recent years, which have shown strong generalization ability. Inspired by the multi-head self-attention (MHSA) mechanism, the graph attention network (GAT) learns weights for different neighbor nodes during training to enhance classification accuracy. Moreover, GAT suits multiple graph structures (Zhao et al. 2022).

Recently, there have been methods proposed for single-scale (CEGCN; Liu et al. 2021) or multi-scale feature extraction using GNNs. Attention multihop graph and multiscale convolutional fusion network (AMGCFN; Zhou et al. 2023), multi-scale receptive fields: Graph attention neural network (MRGAT; Ding et al. 2023) and multiscale graph convolutional network (MSGCN; Zhao et al. 2021) use different hops to achieve multi-scale feature extraction. Though they achieve multi-scale feature extraction, they are based on the same graph space. The effectiveness of feature extraction depends on the initial graph space making use of a node under a single graph space tends to be impure. Once multi-scale features are extracted in different graph spaces, there will inevitably be misaligned superpixel sizes in different graph spaces. Local aggregation and global attention network (LAGAN; Chen et al. 2023) uses two superpixel fusion strategies, namely minimum fusion and maximum fusion, which can fuse two superpixels of different sizes, but at the cost of lose some original pixel-level features and cannot achieve perfect alignment. GACP proposes a method for converting pixel-level features to superpixel-level features, but it can only perform the conversion in a single graph space and cannot achieve multiple graph space transformations. Using dense connections in GNNs can make the extracted features richer, as demonstrated by methods like AMGCFN and Deep Hybrid: Multi-graph neural network (DHMG; Yao, Zhi-Li, et al. 2023). AMGCFN achieves dense connections between different scales, while DHMG achieves dense connections between different feature extraction methods. However, both methods extract features in the same graph space, and achieving dense connections between features extracted from different graph spaces can result in even richer features. After obtaining rich feature maps, further feature fusion can be performed, such as MSFGW and multiple spatial features extraction-fusion (MSFs-EF; Liao and Wang 2020), which fuse the multiple feature maps. For fusing multiple feature maps, methods such as extracting joint spatial-spectral information can be further used (Huang et al. 2022).

Taking inspiration from the notable accomplishments of Spectral-Spatial convolution, GAT, and DenseNet mentioned in HSI classification, we present a pioneering multi-scale densely connected graph attention network tailored to address the limitations of other approaches rooted in the multi-scale concept and dense graph network. This innovative network, known as MSDesGATnet, introduces a novel graph node feature alignment module (GNFA) that enables dense connections across various scales while preserving the original pixel-level features. Simultaneously, it establishes strong associations across different graph spaces. Leveraging the GNFA module and DenseNet, MSDesGATnet outperforms other GNN-based multi-graph methods like the dynamic multi-scale graph convolutional network classifier (DMSGer; Yang et al. 2022). Our final feature maps exhibit exceptional noise resistance and an exponentially vast collection of feature maps, which is evident in both the accuracy of classification and the resultant classification map. The key contributions of this paper are as follows:

1. To combine deep graph node features of different scales and implement a DenseNet architecture, we have developed the Graph Node Feature Alignment modules (GNFA) that utilize two matrix multiplications as spatial transformations at each cross-scale connection. To the best of our knowledge, this is the first instance of integrating a multi-scale graph with DenseNet for HSI classification.

2. SELF and superpixel segmentation are employed for dimensionality reduction and multi-scale graph construction, respectively. Furthermore, it saves memory and is time-consuming.

3. To address the variability of spectral signatures in HSI scenes, we put forward a groundbreaking end-to-end multi-scale graph-dense network integrated with an MHSA mechanism. By leveraging the amalgamated features within the multi-scale graph spaces, we enhance the robustness and accuracy of semi-supervised HSI classification.

4. MSDesGATnet exhibits superior accuracy across various publicly accessible datasets, effectively demonstrating its robustness in the face of node impurities. This unique attribute enables fellow researchers to leverage our framework when exploring graph node feature fusion algorithms with new datasets, eliminating the need for additional time-consuming searches for original datasets with compatible resolutions.

Relevant methods

In this section, we introduce the remaining three algorithms utilized in our proposed approach. For data preprocessing, we employ the Semi-LFDA and simple linear iterative clustering (SILC) algorithms. The former serves to reduce data dimensions, while the latter constructs graph spaces. In the feature iteration phase, we incorporate the GATv2 algorithm (Brody et al. 2021).

Semi-LFDA algorithm

The original hyperspectral image comprises numerous channels, often in the dozens or even hundreds, which may contain considerable noise. Hence, it becomes necessary to diminish the redundant dimensions of the data while retaining the bands that can potentially benefit our classification task. To leverage the advantages of both supervised and unsupervised dimensionality reduction techniques, Semi-LFDA was employed for reducing the data dimensions. In SELF, the labeled data contributes to the projection via LFDA and PCA is used to preserve the global structure of the unlabeled data.

Assuming the existence of labeled samples ${\mathbf{X}}^{\mathbf{L}}\in {\mathbb{R}}^{d\times n\prime }$, unlabeled samples ${\mathbf{X}}^{\mathbf{U}}\in {\mathbb{R}}^{d\times n^{″}}$, $\mathbf{X}=[{\mathbf{X}}^L,{\mathbf{X}}^U],n=n^{\prime }+n^{″}$, then ${\mathbf{X}}^{\mathbf{L}}$ is utilized to compute the local between-class scatter matrix ${\mathbf{S}}^{(\mathit{\text{lb}})}$ and local within-class scatter matrix ${\mathbf{S}}^{(\mathit{\text{lw}})}$ for the labeled samples, and subsequently, the total scatter matrix ${\mathbf{S}}^{(t)}$ was derived. Next, ${\mathbf{S}}^{(\mathit{\text{lb}})}$ and ${\mathbf{S}}^{(\mathit{\text{lw}})}$ was regularized as: [1] $$\left\{\begin{array}{c}{S}^{(\mathit{\text{rlb}})}=(1-\beta )S^{(\mathit{\text{lb}})}+\beta S^{(t)}\\ S^{(\mathit{\text{rlw}})}=(1-\beta )S^{(\mathit{\text{lw}})}+\beta I_d\end{array}\right.$$[1] where $\beta$ determines the balance between PCA and LFDA. To build the transformation matrix, the eigenvectors associated with eigenvalues that contribute to a cumulative sum exceeding a threshold such as 99%, as shown in the following equation: [2] $$\left\{{\lambda }_k,{\phi }_k\right\}\leftarrow S^{(\mathit{\text{rlb}})}{\phi }_k={\lambda }_k{S}^{(\mathit{\text{rlw}})}{\phi }_k$$[2]

These eigenvectors constitute the transformation matrix. We employ it to reduce the data dimensionality, subsequently feeding the processed image into our network alongside the SILC algorithm.

SLIC algorithm

Utilizing the dimensionality-reduced data, we conduct superpixel segmentation on the dataset. Each superpixel resulting from the segmentation is treated as a node, with its feature being the average of the pixel features within that region, as depicted in Figure 1. Substituting pixels with superpixels can significantly reduce computational requirements and complexity. Additionally, MSDesGATnet relies on the SLIC algorithm to repeatedly initialize different numbers of cluster centers to generate a multi-scale graph.

Figure 1. SLIC algorithm for graph construction.

GATv2 algorithm

In GNN, the features of each node are aggregated and updated. To better enable MSDesGAT for feature extraction in multi-scale graph space, we use dynamic Graph attention. Assuming $[{\mathbf{H}}_{\mathbf{1}},{\mathbf{H}}_{\mathbf{2}},\dots {\mathbf{H}}_{\mathbf{n}}]$ represents the features of $n$ nodes, the node features ${\mathbf{H}}_i^k$ are updated according to the following procedure: [3] $${\mathbf{H}}_i^h={\alpha }_{i,i}\mathbf{W}{\mathbf{H}}_i^k+\sum _{j\in \mathcal{N}(i)}{\alpha }_{i,j}\mathbf{W}{\mathbf{H}}_j^k$$[3] $\mathbf{W}$ is a parameter that can be learned, while $\mathcal{N}(i)$ representing the neighboring nodes of a node $i$. The attention coefficient $\alpha$ is calculated as follows: [4] $${\alpha }_{i,j}=\frac{\text{exp}(a^{\top }\mathit{\text{LeakyReLU}}(W\left[H_i^k\parallel H_j^k\right]))}{{\displaystyle \sum _{j\in \mathcal{N}(i)\cup \left\{i\right\}}\mathit{\text{exp}}(a^{\top }\mathit{\text{LeakyReLU}}(W\left[H_i^k\parallel H_j^k\right]))}}$$[4]

To ensure numerical stability and robustness, the attention coefficients were normalized and the attention mechanism was extended to multi-head attention.

By employing the MHSA mechanism, GATv2 accomplishes the update of node features at the superpixel level. In our experiments, each single-scale graph undergoes two sequential GATv2 layers. The first layer consists of 4 multi-head attention, while the second layer involves a single attention, as illustrated in Figure 2.

Figure 2. Composition of GAT layer in MSDesGATnet.

Proposed MDesGATnet framework

In this section, we will present our proposed MSDesGATnet architecture for hyperspectral classification, as depicted in Figure 3. It comprises two main components: Graph Node Feature Alignment modules and Fusion feature modules. The former facilitates the transformation of data between graph space and pixel space, while the latter connects the data processed by GATv2 to the input and iterates this process, similar to DenseNet. Additionally, we incorporate spectral-spatial convolution to fuse the rich feature maps obtained from multiple scales.

Figure 3. Flowchart of the proposed MSDesGATnet.

The input to the entire MSDesGATnet is derived from the image data after SELF dimensionality reduction. Let $\mathbf{X}\in {\mathbb{R}}^{n\times r}$ represent the reduced-dimensional data, which consists of $n$ pixels, with each pixel containing $r$ dimensional features.

Graph node feature alignment modules

When employing multi-scale graph neural networks for feature extraction, it is crucial to address the transfer of node features across different scale graphs. As the boundaries of superpixel segmentations at different scales often do not align precisely, it is not feasible to directly fuse node features across scales. Therefore, we propose the GNFA module, which enables the transfer of node features through a sequential transformation at the pixel-node-pixel level.

The relationship between nodes and pixels can be represented by the transformation matrix $\mathbf{Q}\in {\mathbb{R}}^{n\times m}$. For instance, if we obtain m nodes through superpixel segmentation and there is a total of $n$ pixels, we can construct the $\mathbf{Q}$ matrix as follows: [5] $$Q_{\mathit{\text{nm}}}=\left\{\begin{array}{cc}1& \text{\hspace{0.17em}if\hspace{0.17em}}{x}_n\in V_m\\ 0& \text{\hspace{0.17em}else\hspace{0.17em}}\end{array}\right.$$[5] where ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots {\mathbf{x}}_n$ represents the sequence of pixels and ${\mathbf{V}}_1,{\mathbf{V}}_2,\dots {\mathbf{V}}_m$ represents the sequence of nodes. This equation demonstrates that if a pixel ${\mathbf{x}}_n$ belongs to the nodes ${\mathbf{V}}_m$, then the corresponding value in the $\mathbf{Q}$ matrix at position $(n,m)$ is 1; otherwise, it is 0. As depicted in Figure 4, the node ${\mathbf{V}}_2$ includes ${\mathbf{x}}_3,{\mathbf{x}}_4,{\mathbf{x}}_{11}{\mathbf{x}}_{12},{\mathbf{x}}_{13}$, so the corresponding value in the column of ${\mathbf{V}}_2$ in $\mathbf{Q}$ is 1, while the remaining values are 0. Similarly, the values in each column of $\mathbf{Q}({\mathbf{V}}_1,{\mathbf{V}}_2,\dots {\mathbf{V}}_n)$ can be determined based on the included pixels.

Figure 4. Acquire the values of the Q matrix.

Assuming $\mathbf{X}$ represents the pixel-level space and $\mathbf{H}\in {\mathbb{R}}^{\mathbf{m}\times \mathbf{r}}$ represents the node-level space, we can convert the pixel-level space into the node-level graph space as an encoding step: [6] $$H={\widehat{Q}}^{\top }X$$[6]

$\top$ denotes the transpose operator. $\widehat{Q}$ serves as the regularization for $\mathbf{Q}$, as the node feature is defined as the average of the pixels associated with the node: [7] $$\stackrel{\mathbf{⁁}}{\mathbf{Q}}=\mathbf{Q}{\mathbf{D}}_{\mathbf{Q}}^{-\mathbf{1}}$$[7] ${\mathbf{D}}_{\mathbf{Q}}$ represents the diagonal matrix, and its inverse matrix ${\mathbf{D}}_{\mathbf{Q}}^{-\mathbf{1}}\in {\mathbb{R}}^{m\times m}$ is defined as follows: [8] $${\mathbf{D}}_{\mathbf{Q}}^{-\mathbf{1}}=\mathit{\text{diag}}({(\sum _j{\mathbf{Q}}_{\mathbf{j}\mathbf{1}})}^{-1}\cdots {(\sum _j{\mathbf{Q}}_{\mathbf{\text{jm}}})}^{-1}))$$[8]

Similar to Equation (6), the $\mathbf{Q}$ matrix can also be used as a decoding step to transform node-level features back to pixel-level features: [9] $${\mathbf{X}}^{\mathbf{\prime }}=\mathbf{Q}(\mathit{\text{GATv}}2(\mathbf{H},\mathbf{A}))$$[9] where $\mathit{\text{GATv}}2()$ represents the output of the GAT layer, and the GAT layer updates the node features $\mathbf{H}$. The adjacency matrix $\mathbf{A}$ represents the current scale graph structure. The output of the GAT layer is multiplied on the left by the transformation matrix $\mathbf{Q}$ to obtain the pixel-level features ${\mathbf{X}}^{\mathbf{\prime }}$.

The GNFA module can be constructed by utilizing the encoding and decoding steps explained earlier. Figure 5 offers an intuitive visualization of the procedure for altering pixel features within a GNFA module. Initially, the node feature is determined by taking the average value of the pixel features during the encoding step. Subsequently, the node feature undergoes aggregation and updating through two GAT layers. Finally, the updated node features are substituted with the pixel-level features during the decoding step. To align two sets of node features from different scale graphs, two GNFA modules can be concatenated together. In our MSDesGATnet, the first GNFA module is employed for the larger-scale graph, while the second GNFA module is used for the smaller-scale graph. This approach converts the node features of the larger-scale graph into pixel-level features through the decoding step of the first GNFA module, and subsequently re-encodes the pixel-level features, embedding latent information from the larger-scale graph, into the node features of the smaller-scale graph.

Figure 5. Graph node feature alignment modules.

The SLIC algorithm through the initial seed count can determine the number of nodes. To enhance efficiency, we can pre-compute transformation matrices ${\mathbf{Q}}_k$ and ${\widehat{\mathbf{Q}}}_k$ for each scale $k$, as illustrated in Figure 6. These pre-computed matrices, ${\mathbf{Q}}_k$ and ${\widehat{\mathbf{Q}}}_k$, enable us to construct the GNFA modules in our MSDesGATnet more efficiently.

Figure 6. Qs achieved in multi-scales.

Multiple graphs dense connection network

In this research, we incorporate the DenseNet concept into our multiple graph-dense connection networks to effectively extract multi-scale features. Similar to DenseNet, we can obtain deep features with a larger receptive field while preserving shallow features by stacking multiple GNN layers. Specifically, as depicted in Figure 7, ${\mathbf{X}}_k$ denotes the input of the $k$th GNFA module, and ${\mathbf{X}}_{k+1}$ can be obtained as follows: [10] $${\mathbf{X}}_{k+1}=({\mathbf{Q}}_k{\mathit{\text{GAT}}}_{v2}({\stackrel{\mathbf{⁁}}{\mathbf{Q}}}_k{}^{\top }{\mathbf{X}}_k,{\mathbf{A}}_k))\parallel {\mathbf{X}}_k$$[10]

Figure 7. Multiple graphs dense connection network.

${\mathbf{Q}}_k$ and ${\widehat{\mathbf{Q}}}_k$ are obtained through SLIC segmentation, and ${\mathbf{A}}_k$ is defined. These matrices can be pre-computed and stored in GPU memory to ensure efficient training. In the GNFA modules, we maintain the same dimensions for the input and output. As a result, the feature dimension of ${\mathbf{X}}_{k+1}$ is twice that of ${\mathbf{X}}_k$. In the MSDesGATnet, we utilize four scale GNFA modules, allowing us to obtain high-dimensional features with a feature dimension $2^4=16$ times that of the original.

Spectral-spatial convolution module

In MSDesGATnet, the Spectral-Spatial convolution (SSConv) module is utilized twice. The first time, this module is used to extract shallow features at the pixel level, while the second time, it is employed to fuse the high-dimensional features obtained from the multi-scale GNN. As depicted in Figure 8, the SSConv module comprises two sequential submodules: the spectral convolution submodule and the spatial convolution submodule.

Figure 8. Spectral-Spatial convolution for multi-scales future fusion.

In the spectral convolution submodule, the input undergoes batch normalization as the first step. Then, a 1 × 1 convolution is applied to extract features in the channel dimension. Lastly, a LeakyReLU activation layer is employed for non-linear mapping. As for the spatial convolution submodule, two consecutive 3 × 3 depth-wise convolutions are utilized, with a LeakyReLU layer in between.

Exponential feature maps under the permutation graph space

In our algorithm framework, we have multiple graph spaces, namely ${\mathcal{G}}_1$ and ${\mathcal{G}}_2$. The data unit, $\mathbf{X}\in {\mathbb{R}}^{H\times W\times C}$, is defined as the tensor size of the original dataset. The output results of each GNFA module can be expressed as integer multiples of $\mathbf{X}$. We utilize MHSA to extract features from $\mathbf{X}$ within the graph space ${\mathcal{G}}_x$, denoted as $\mathbf{X}({\mathcal{G}}_x)$. Upon analysis, it has been determined that when $\mathbf{X}$ sequentially passes through two graph spaces, ${\mathcal{G}}_1$ and ${\mathcal{G}}_2$, under the influence of GNFA, the output includes $\mathbf{X}$, $\mathbf{X}({\mathcal{G}}_1)$, $\mathbf{X}({\mathcal{G}}_2)$, and $\mathbf{X}({\mathcal{G}}_1|{\mathcal{G}}_2)$. Furthermore, when analyzing the output of three consecutive graph spaces (adding ${\mathcal{G}}_3$), we obtain even more diverse output feature maps: $\mathbf{X}\mapsto \mathbf{X}({\mathcal{G}}_3)$, $\mathbf{X}({\mathcal{G}}_1)\mapsto \mathbf{X}({\mathcal{G}}_1|{\mathcal{G}}_3)$, $\mathbf{X}({\mathcal{G}}_2)\mapsto \mathbf{X}({\mathcal{G}}_2|{\mathcal{G}}_3)$, $\mathbf{X}({\mathcal{G}}_1|{\mathcal{G}}_2)\mapsto \mathbf{X}({\mathcal{G}}_1\left|{\mathcal{G}}_2\right|{\mathcal{G}}_3)$. Through the dense net operation, the final result can be presented as: [11] $${\mathbf{X}}_{k+1}=\parallel \mathbf{X}(\mathcal{G}\in \{{\mathcal{G}}_1,{\mathcal{G}}_2,\dots {\mathcal{G}}_k\})$$[11]

Equation (11) indicates that the feature maps are processed from the sub-graph spaces of all graph spaces. At the same time, the GNFA module ensures that node features in one graph space are represented in nodes of other graph spaces: [12] $${\mathbf{H}}^{{\mathcal{G}}_x}=\mathit{\text{GNFA}}({\mathbf{Q}}_x,{\mathbf{Q}}_y,{\mathbf{H}}^{{\mathcal{G}}_x})={\widehat{\mathbf{Q}}}_y^{\top }{\mathbf{Q}}_x{\mathbf{H}}^{{\mathcal{G}}_x}$$[12]

Overall, the GNFA module and dense connections equip MSDesGATnet to generate an output that encompasses exponentially increasing rich features (all permutations of $k$ graph spaces). This implies that even if some superpixels in a certain space are impure, their influence on the final output feature map is relatively minor. Therefore, compared to other methods based on a single space, it possesses a stronger anti-interference capability. Moreover, SSConv integrates these abundant feature maps, and a weighted loss function tackles the issue of an imbalanced number of ground-truth categories. Table 1 presents the pseudo-code of MSDesGATnet.

Table 1. Pseudo code of MSDesGATnet.

MSDesGATnet Algorithm

Input: Processed image by SELF and SSConv; Multi-scales=$m$ Output: Classification results Flatten the image into $X\in {ℝ}^{n\times r}$; for each $k\in [1,m]$ do       Encode the input $X_k$ into $H_k={\widehat{Q}}_k^{\top }{X}_k$       $H_{k^{\prime }}=\mathit{\text{GATv}}2(H_k,A_k)$       Decode $H_{k^{\prime }}$ back to $X_{k^{\prime }}=Q_k{H}_{k^{\prime }}$ ($k$th GNFA module)       ${\mathbf{X}}_{\mathbf{k}+\mathbf{1}}=({\mathbf{Q}}_{\mathbf{k}}{\mathit{\text{GAT}}}_{v2}({\stackrel{\mathbf{⁁}}{\mathbf{Q}}}_{\mathbf{k}}{}^{\top }{\mathbf{X}}_{\mathbf{k}},{\mathbf{A}}_{\mathbf{k}}))\parallel {\mathbf{X}}_{\mathbf{k}}$ end for DenseNet output $X_{m+1}\in {ℝ}^{n\times 2^{m}r}$ Apply SSConv to fuse ${\mathbf{X}}_{\mathbf{m}+\mathbf{1}}$ to get $\mathbf{F}$ Add a FC layer to obtain ${\mathbf{F}}^{\mathbf{\prime }}\in {\mathbb{R}}^{h\times w,\mathit{\text{categories}}}$ Softmax Return Classification map

Experiments and analysis

In this section, we choose four public hyperspectral datasets for classification to validate the effectiveness of our proposed MSDesGATnet, namely Indian pines (IP), Kennedy Space Center (KSC), Pavia University (PU), and WHU-Hi-HongHu. Additionally, we select five state-of-the-art algorithms for comparison. We also conduct ablation studies and multi-scale sensitivity analyses.

Dataset description

IP dataset

The Indian Pines dataset was captured by AVIRIS (0.4–2.5 μm) in the USA. The shape of the HSI is 145 × 145 and it comprises 220 bands. The ground truth of the Indian Pines has 16 categories. Figure 9 displays the false color and ground truth of Indian Pines, respectively.

Figure 9. IP: (a) false color map (b) ground truth.

KSC dataset

The KSC dataset was captured by the AVIRIS sensor at the Kennedy Space Center in Florida on March 23, 1996. After the removal of water vapor noise, 176 bands remain, with a spatial resolution of 18 meters. The dataset comprises a total of 13 categories and 614 × 512 pixels. Figure 10 displays the false color and ground truth, respectively.

Figure 10. KSC: (a) false color map (b) ground truth.

PU dataset

The Pavia University dataset is a portion of the hyperspectral dataset of the image of Pavia, Italy, captured by the German airborne reflection optical spectral imager in 2003. The imager continuously captures 115 bands in the 0.43–0.86 μm wavelength range. Due to noise, 12 of these bands are eliminated, so the image generally used consists of the remaining 103 spectral bands. The image size is 610 × 340, and the ground truth contains nine categories. Figure 11 displays the false color and ground truth of PU, respectively.

Figure 11. PU: (a) false color map (b) ground truth.

WHU-Hi-HongHu dataset

The WHU-Hi-HongHu dataset was gathered by a DJI Matrice 600 Pro drone in Wuhan, Hubei Province, China. It encompasses 22 distinct land cover types, including different varieties of crops such as Chinese cabbage and cabbage. The images measure 940 × 475 pixels and comprise 270 channels ranging from 200 to 1000 nanometers in wavelength. The spatial resolution is 0.043 meters, classifying it as a high-resolution (H²) hyperspectral dataset. Figure 12 presents the false color and ground truth, respectively.

Figure 12. WHU-Hi-HongHu: (a) false color map (b) ground truth.

Experimental results and quantitative discussion

Experimental settings

The symbols of MSDesGATnet’s hyper-parameters are defined as follows: $\mathbf{M}$ represents multi-scale parameters; $\mathit{\text{lr}}$ stands for learning rate; $\mathbf{L}$ denotes the number of the GNFA modules; $\mathbf{T}$ signifies the number of training epochs. We set different hyper-parameters for three datasets, as shown in Table 2. After our experiments on multiple datasets, we found that using MSDesGAT at scale 3 or so could makes the combined metric of accuracy-time optimal.

Table 2. The hyperparameter settings of MSDesGAT.

Dataset	$\mathbf{M}$	$\mathbf{L}$	$\mathit{\text{lr}}$	$\mathbf{T}$
IP	[1400, 1200, 1000, 800]	4	1e^-3	700
PU	[1400. 1200, 1000, 800]	4	1e^-3	700
KSC	[1600, 1400, 1200, 1000]	4	1e^-3	700
HongHu	[800, 600, 200]	3	1e^-3	700

For fairness, all comparison methods and MSDesGATnet are based on the data after SELF. The number of channels after SELF is 95 for IP, 6 for PU, 106 for KSC, and 109 for WHU-Hi-HongHu, respectively.

All the experimental algorithms are implemented with Python-3.9, PyTorch-1.12.1, and Adam optimizer with an RTX-3090 GPU.

Cross-validation and comparison methods

For the dataset selection, 25 samples of each class under each dataset were selected as the training set. We’re using these samples in a 5-fold cross-validation method. In the 5-CV, the results of each iteration are shown in the table. The test set includes all labeled samples, and the data presented for the test set are the averages of the 5-fold cross-validation results, along with the standard deviation.

To assess the performance of MSDesGATnet, we’re comparing it with six other state-of-the-art methods. These methods include three CNN-based methods: (1) Dynamic Convolution Neural Network (DCNN; Makantasis et al. 2015), (2) Double-Branch Dual-Attention Network (DBDAnet; Li et al.2020), (3) Spectral-Spatial Residual Network (SSRN; Zhong et al.2018), and two GNN-based methods: (1) Edge Labeling Graph Neural Network (EGNN; Hu et al.2022), (2) Attention Multihop Graph and Multiscale Convolutional Fusion Network (AMGCFN; Zhou et al. 2023). We’re using three evaluation metrics in our experiments, namely, overall accuracy (OA), average accuracy (AA), and Kappa ($\kappa$).

The quantitative and visual results of IP

Table 3 provides quantitative experimental results on the Indian Pines dataset, including the precision of each class, along with the corresponding OA, AA, and $\kappa$. We’ve highlighted the maximum accuracy in each row. The results show that the OA, $\kappa$, and AA of MSDesGATnet were higher than those of other methods, and the standard deviation was relatively small. We can find that GNN-based methods perform similarly to CNN-based methods, whereas at the same time, it is with multi-scale methods such as AMGCFN and our proposed MSDesGAT that performs better compared to the single-scale method EGNN, even with the presence of a large number of continuously distributed crops in the IP dataset. Comparing AMGCFN and MSDesGAT, MSDesGAT improves the AA by about 10%, and although both methods use multiple scales, the number of feature maps extracted by MSDesGAT is exponential, and the final feature fusion module can further fuse richer features than AMGCFN. Based on Figure 14, we also found that MSDesGATnet can also classify similar spectral features with high accuracy, such as C5, C6, and C7. This is due to the attention mechanism of MSDesGAT, which achieves similar effects as DBDAnet and AMGCFN.

Table 3. OAs, AAs and Kappas obtained by all methods in IP. The bolded value in each row indicates the best performance.

No.	Class	Labeled	DCNN	DBDAnet	SSRN	EGNN	AMGCFN	MSDesGAT
1	Alfalfa	46	99.13 ± 1.19	100.0 ± 0.0	100.0 ± 0.0	98.7 ± 1.19	100.0 ± 0.0	100.0 ± 0.0
2	Corn-notill	1428	69.58 ± 10.08	76.02 ± 6.14	82.34 ± 10.79	58.0 ± 9.3	71.74 ± 12.23	89.02 ± 9.01
3	Corn-mintill	830	77.59 ± 3.41	87.59 ± 3.66	89.4 ± 7.35	62.07 ± 6.86	83.25 ± 7.22	92.7 ± 2.97
4	Corn	237	93.33 ± 3.51	99.49 ± 1.13	98.4 ± 3.59	94.18 ± 1.5	95.44 ± 4.26	100.0 ± 0.0
5	Grass-pasture	483	90.06 ± 3.75	90.81 ± 1.69	92.8 ± 2.46	87.49 ± 1.75	91.76 ± 4.27	94.53 ± 4.09
6	Grass-trees	730	94.3 ± 2.74	98.05 ± 0.67	99.21 ± 0.64	90.9 ± 5.95	92.08 ± 3.72	97.89 ± 1.99
7	Grass-mowed	28	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0
8	Hay-windrowed	478	97.41 ± 1.59	100.0 ± 0.0	100.0 ± 0.0	96.49 ± 2.43	98.62 ± 2.15	99.46 ± 1.0
9	Oats	20	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0
10	Soybean-notill	972	81.15 ± 9.43	85.95 ± 5.61	91.95 ± 8.86	72.53 ± 7.53	80.1 ± 8.55	92.37 ± 4.82
11	Soybean-mintill	2455	54.04 ± 7.99	0.33 ± 0.75	45.11 ± 14.44	53.16 ± 8.13	69.63 ± 20.52	85.28 ± 5.08
12	Soybean-clean	593	86.17 ± 2.11	88.4 ± 4.2	94.87 ± 2.42	78.04 ± 12.17	74.47 ± 11.74	90.59 ± 4.45
13	Wheat	205	98.93 ± 0.87	99.71 ± 0.65	100.0 ± 0.0	98.63 ± 1.7	97.76 ± 4.02	98.54 ± 3.01
14	Woods	1265	88.85 ± 6.72	87.35 ± 7.63	94.32 ± 4.11	89.2 ± 4.13	89.91 ± 6.91	90.4 ± 4.96
15	Buildings-Grass	386	90.73 ± 4.17	99.84 ± 0.23	98.86 ± 1.18	92.85 ± 1.58	99.48 ± 0.55	99.74 ± 0.37
16	Stone-Steel	93	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	97.85 ± 3.65	99.14 ± 1.18	99.57 ± 0.96
	OA(%)		77.45 ± 2.16	67.62 ± 1.58	81.3 ± 3.31	72.79 ± 1.51	81.63 ± 4.79	91.42 ± 1.9
	AA(%)		88.83 ± 1.14	88.35 ± 1.0	92.95 ± 1.09	85.63 ± 1.54	90.21 ± 1.99	95.63 ± 0.7
	Kappa		74.81 ± 2.34	64.75 ± 1.73	79.19 ± 3.58	69.41 ± 1.72	79.33 ± 5.19	90.28 ± 2.13
FLOPs(G)			0.052	6.91	6.11	0.937	2.046	1.304
Param(M)			0.497	0.382	0.364	0.051	0.342	0.062
Overall Accuracy in 5-CV
1	77.09	65.0	80.5	74.8	85.5	90.1
2	75.7	67.5	82.2	73.4	81.5	89.8
3	79.4	68.0	85.3	70.7	79.7	94.3
4	75.7	68.3	76.2	72.1	86.6	92.2
5	79.9	69.1	81.9	72.7	74.7	90.6
Average Accuracy in 5-CV
1	88.14	86.6	92.3	86.4	91.8	95.1
2	88.64	88.3	93.2	87.5	90.7	95.2
3	90.0	88.6	94.7	85.0	87.5	96.8
4	87.4	88.9	92.0	83.4	92.1	95.5
5	89.8	89.1	92.3	85.6	88.7	96.4

Figure 13 demonstrates the classification map of all methods. Compared with the Ground Truth, we observe that the classification map achieved by MSDesGATnet is closer and smoother to the Ground truth due to the multiple graph encoding and decoding processes. The classification results of MSDesGAT are more in line with the practical needs, i.e., the crops in the IP dataset are spatially continuous within classes and discrete between classes. Therefore, it is reasonable to infer that MSDesGAT performance on the IP baseline dataset is more accurate and robust.

Figure 13. Classification maps of all methods in Indian pines. (a) false-color map. (b) Ground truth. (c) DCNN (OA=71.03%) (d) DBDAnet (OA=65.01%) (e) SSRN (OA=76.27%) (f) EGNN(OA=70.76%) (g) AMGCFN (OA=72.4%) (h) MSDesGATnet (OA=94.3%).

The quantitative and visual results of KSC

Similar to the findings in Table 4, MSDesGATnet boasts the highest OA on the KSC dataset. Although AMGCFN also achieved a high AA, the results didn’t show a significant difference. However, MSDesGATnet demonstrated superior accuracy across more categories. At the same time, the performance of the CNN-based method and the GNN method are both excellent due to the small number of samples that have been labeled, 25 samples selected account for 10% to 2% of the total samples, and at the same time, the samples of each class are centrally distributed, which is conducive to convolution kernel capturing by the CNN method.

Table 4. OAs, AAs and Kappas obtained by all methods in KSC. The bolded value in each row indicates the best performance.

No.	Class	Labeled	DCNN	DBDAnet	SSRN	EGNN	AMGCFN	MSDesGAT
1	Srub	758	89.72 ± 4.91	99.13 ± 0.78	93.98 ± 4.5	88.02 ± 8.42	99.89 ± 0.24	99.5 ± 0.56
2	Willow swamp	250	91.52 ± 4.82	97.7 ± 3.05	97.61 ± 2.9	87.08 ± 4.75	98.27 ± 2.58	95.56 ± 2.32
3	CP hammock	262	90.78 ± 3.51	98.28 ± 2.35	94.45 ± 6.8	94.14 ± 4.48	98.12 ± 2.71	99.77 ± 0.52
4	Slash pine	258	68.17 ± 11.57	94.21 ± 6.39	94.92 ± 10.0	86.83 ± 6.52	97.14 ± 2.5	96.43 ± 2.53
5	Oak/Broadleaf	176	78.01 ± 5.35	96.27 ± 3.29	94.78 ± 7.4	96.89 ± 6.94	99.88 ± 0.28	97.89 ± 3.42
6	Hardwood	237	82.79 ± 4.47	100.0 ± 0.0	97.99 ± 1.7	99.04 ± 1.09	100.0 ± 0.0	99.62 ± 0.85
7	Swamp	126	98.48 ± 1.44	97.33 ± 5.45	97.9 ± 3.33	100.0 ± 0.0	100.0 ± 0.0	99.3 ± 0.98
8	Graminoid	423	89.74 ± 4.49	99.35 ± 1.45	97.59 ± 5.2	81.67 ± 11.79	98.84 ± 2.59	100.0 ± 0.0
9	Spartina marsh	499	91.23 ± 5.43	98.85 ± 1.67	89.92 ± 6.8	95.12 ± 5.83	99.81 ± 0.43	100.0 ± 0.0
10	Cattail marsh	396	94.21 ± 3.09	99.8 ± 0.44	98.51 ± 1.7	98.17 ± 1.41	100.0 ± 0.0	100.0 ± 0.0
11	Salt marsh	408	98.14 ± 2.03	99.43 ± 0.8	99.33 ± 1.4	97.61 ± 0.88	100.0 ± 0.0	100.0 ± 0.0
12	Mud marsh	484	88.55 ± 2.38	100.0 ± 0.0	96.22 ± 2.5	95.9 ± 3.34	96.26 ± 4.55	99.68 ± 0.71
13	Water	866	99.78 ± 0.48	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0
	OA(%)		91.18 ± 1.99	99.0 ± 0.41	96.46 ± 1.3	93.85 ± 1.15	99.19 ± 0.71	99.34 ± 0.15
	AA(%)		89.32 ± 1.78	98.49 ± 0.57	96.4 ± 1.58	93.88 ± 0.7	99.09 ± 0.75	99.0 ± 0.31
	Kappa		90.2 ± 2.2	98.89 ± 0.46	96.07 ± 1.5	93.17 ± 1.27	99.1 ± 0.79	99.27 ± 0.16
FLOPs(G)			0.381	6.06	5.370	12.04	14.982	20.72
Parm(M)			3.582	0.338	0.327	0.059	0.318	0.066
Overall Accuracy in 5-CV
1	92.9	99.3	96.1	92.1	99.6	99.2
2	88.2	99.5	94.5	94.6	98.6	99.4
3	92.7	98.8	96.2	93.4	98.2	99.1
4	91.8	98.5	98.3	93.9	99.9	99.4
5	90.1	98.6	97.0	95.0	98.6	99.4
Average Accuracy in 5-CV
1	91.1	99.2	96.3	93.0	99.4	98.7
2	86.5	98.5	94.0	94.3	98.5	99.3
3	90.4	98.7	95.9	93.3	98.0	98.7
4	89.6	97.9	98.3	93.9	99.9	98.8
5	88.8	97.9	97.2	94.7	97.02	99.3

Figure 14 presents the detailed classification outcomes of MSDesGATnet. Misclassification occurred between C4 and C5 and between C2 and C3. Ground truth reveals that C4/C5 and C2/C3 are in close proximity to each other and have small targets. Facing the problem of superpixel impurity, AMGCFN mitigates this problem by introducing a CNN module, and MSDesGAT mitigates this problem by a densely connected multi-scale graph space-feature fusion method, so that AMGCFN has the strongest discriminative ability on C4/C5, and MSDesGAT has the strongest discriminative ability on C2/C3, but at the same time, the number of parameters(M) of MSDesGAT is much smaller than that of AMGCFN and FLOPs(G) are on the same order of magnitude as AMGCFN.

Figure 14. Normalized confusion matrix of MSDesGATnet on (a) IP (b) KSC.

Figure 15 displays the classification maps, with particular emphasis on the two magnified view regions within each map. For the upper magnified region, it can be observed that the classification results of GNN-based method are smoother than those CNN-based method, and MSDesGAT has fewer misclassifications. For the lower magnified region, it can be observed that the classification results of AMGCFN and the proposed MSDesGAT are closer to GT than all other methods. However, in the misclassification results of AMGCFN, there is a misclassification of C3 to the distant C1. MSDesGAT, while having a global receptive field, alleviates this impact through a multi-scale dense graph spatial strategy, which is more in line with practical scenarios.

Figure 15. Classification maps of all methods on KSC dataset. (a) Ground truth. (b) DCNN(OA=88.23%) (c) DBDAnet (OA=98.5%) (d) SSRN (OA=94.53%) (e) EGNN (OA=92.13%) (f) AMGCFN (OA=98.25%) (g) MSDesGATnet (OA=99.46%). In (a)-(g), zoomed-in views of regions of bule were shown in above and bottom at each classification map.

The quantitative and visual results of PU

Table 5 presents the accuracy of the Pavia University dataset, where it’s observed that our proposed MSDesGATnet achieves the highest OA. When compared to other methods such as AMGCFN, MSDesGATnet not only exhibits significant generalization performance on large-scale categories with large samples like C2 but also on small-scale categories like C8. Concurrently, it’s noted that the subpar performance of EGNN and DCNN is due to the small sample size and varying size of the features. While SSRN can extract spatial and spectral features from relatively clustered samples, both AMGCFN and MSDesGATnet demonstrated commendable performance using multi-scale feature extraction and attention mechanisms.

Table 5. OAs, AAs and Kappas obtained by all methods in PU. The bolded value in each row indicates the best performance.

No.	Class	Labeled	DCNN	DBDAnet	SSRN	EGNN	AMGCFN	MSDesGAT
1	Asphalt	6631	67.84 ± 6.92	91.18 ± 7.66	91.96 ± 2.46	91.52 ± 2.54	77.44 ± 12.08	72.8 ± 7.86
2	Meadows	18649	33.11 ± 14.73	60.43 ± 12.42	77.14 ± 4.38	63.92 ± 11.51	88.64 ± 4.19	91.53 ± 4.5
3	Gravel	2099	85.26 ± 4.99	92.73 ± 6.97	86.64 ± 7.98	86.85 ± 8.48	97.48 ± 3.4	96.35 ± 5.25
4	Trees	3064	96.91 ± 1.78	97.44 ± 1.32	98.71 ± 1.62	91.8 ± 3.8	71.48 ± 15.6	68.47 ± 3.92
5	Painted metal sheets	1345	99.02 ± 1.4	99.85 ± 0.12	100.0 ± 0.0	97.07 ± 2.96	99.73 ± 0.6	95.67 ± 2.96
6	Bare soil	5029	93.66 ± 4.25	99.73 ± 0.5	98.1 ± 2.7	88.68 ± 6.33	94.36 ± 6.07	98.99 ± 0.26
7	Bitumen	1330	92.18 ± 1.98	99.64 ± 0.53	99.05 ± 1.41	97.62 ± 3.3	93.1 ± 5.54	97.64 ± 5.03
8	Self-Blocking bricks	3682	72.25 ± 6.93	84.12 ± 20.4	88.06 ± 12.71	89.07 ± 2.87	84.24 ± 2.13	95.28 ± 2.27
9	Shadows	947	99.28 ± 0.76	99.54 ± 0.24	99.96 ± 0.09	95.46 ± 3.15	90.62 ± 6.3	79.81 ± 8.47
	OA(%)		61.48 ± 6.92	79.42 ± 5.75	86.76 ± 2.62	79.18 ± 5.56	86.93 ± 4.51	88.47 ± 1.16
	AA(%)		82.17 ± 1.93	91.63 ± 1.86	93.29 ± 1.2	89.11 ± 1.87	88.57 ± 4.25	88.5 ± 1.29
	Kappa		55.09 ± 6.95	74.84 ± 6.42	83.24 ± 3.19	74.28 ± 6.33	83.09 ± 5.72	84.96 ± 1.42
FLOP(s)			0.0005	4.414	3.129	7.899	29.076	0.113
Parm(M)			0.005	0.202	0.216	0.046	0.338	0.0005
Overall Accuracy in 5-CV
1	52.81	70.46	88.63	71.45	90.18	88.36
2	60.16	85.99	83.68	85.84	81.57	89.79
3	58.77	81.6	86.06	79.98	82.98	89.41
4	64.23	80.98	85.23	82.39	87.99	87.85
5	71.45	78.05	90.18	76.26	91.95	86.93
Average Accuracy in 5-CV
1	79.65	90.17	94.3	86.71	93.61	88.88
2	82.21	94.14	92.02	91.85	84.65	86.21
3	81.99	93.06	93.21	89.35	83.73	89.27
4	81.91	89.99	92.23	89.35	89.64	89.04
5	85.08	90.78	94.7	88.28	91.21	89.12

Figure 17 presents the specific classification outcomes of MSDesGATnet on PU, while Figure 16 illustrates the classification map of all methods. It’s observed that the misclassification rates between C4 and C1, as well as between C9 and C5, are relatively higher. This is attributed to these categories being near smaller targets, limiting the effectiveness of the attention mechanism under multi-scale conditions. However, the classification results for C7 and C6 outperform other methods like AMGCFN.

Figure 16. Classification maps of all methods on PU dataset. (a) Ground truth. (b) DCNN (OA = 52.80%) (c) DBDAnet (OA = 70.46%) (d) SSRN (OA = 83.68%) (e) EGNN (OA = 71.45%) (f) AMGCFN (OA = 84.04%) (g) MSDesGATnet (OA = 89.79%).

The quantitative and visual results of WHU-Hi-HongHu

Table 6 showcases the quantitative classification accuracies of all methods on the HongHu dataset. It’s observed that our proposed MSDesGATnet achieved the highest AA, albeit slightly lower than AMGCFN in terms of OA and kappa. MSDesGATnet also excelled in more categories and exhibited relatively small standard deviations, with notably higher classification accuracies on C3 and C14 compared to other methods. Additionally, it was found that the performance of CNN-based methods was inferior to that of GNN-based methods. This is attributed to the fact that extracting a small number of labeled samples from a large set can cause significant separation between training samples beyond the receptive field of CNNs. However, GNN-based methods, through multi-layer information propagation, can aggregate global information and are more suitable for training with a small number of samples than CNN-based methods.

Table 6. OAs, AAs and Kappas achieved by all methods in WHU-Hi-Honghu. The bolded value in each row indicates the best performance.

No.	Class	Labeled	DCNN	DBDAnet	SSRN	EGNN	AMGCFN	MSDesGAT
1	Red roof	14041	89.14 ± 2.17	89.98 ± 4.18	84.4 ± 12.37	77.81 ± 4.78	95.31 ± 3.58	96.39 ± 4.1
2	Road	3512	88.31 ± 4.4	93.5 ± 4.83	92.47 ± 3.77	84.19 ± 6.89	92.75 ± 3.4	74.67 ± 24.18
3	Bare soil	21821	68.29 ± 3.78	77.94 ± 7.48	81.57 ± 3.82	81.36 ± 6.47	86.57 ± 2.15	92.42 ± 5.19
4	Cotton	163285	11.83 ± 8.44	0.0 ± 0.0	0.0 ± 0.0	77.57 ± 4.95	95.97 ± 1.8	89.86 ± 4.33
5	Cotton firewood	6218	89.21 ± 3.76	94.24 ± 4.74	88.45 ± 3.74	95.35 ± 6.1	96.3 ± 2.49	99.13 ± 0.4
6	Rape	44557	83.78 ± 3.72	75.23 ± 7.34	79.84 ± 9.39	84.57 ± 4.49	91.87 ± 1.66	90.73 ± 2.96
7	Chinese cabbage	24103	60.9 ± 3.8	62.58 ± 10.85	49.04 ± 18.29	68.44 ± 3.7	73.22 ± 6.02	63.82 ± 2.16
8	Pakchoi	4054	60.96 ± 3.6	85.46 ± 8.22	69.22 ± 12.21	99.98 ± 0.02	94.09 ± 3.52	99.43 ± 1.28
9	Cabbage	10819	92.82 ± 2.55	93.46 ± 2.95	92.45 ± 5.2	94.27 ± 1.32	93.7 ± 1.9	94.66 ± 0.95
10	Tuber mustard	12394	67.04 ± 5.88	84.97 ± 4.98	75.03 ± 8.1	85.57 ± 3.95	83.31 ± 9.51	84.63 ± 2.51
11	Brassica parachinensis	11015	54.05 ± 5.62	86.11 ± 4.95	56.8 ± 14.08	77.47 ± 3.29	77.54 ± 1.35	85.95 ± 6.95
12	Brassica chinensis	8954	70.51 ± 4.29	81.61 ± 8.23	71.24 ± 7.86	85.6 ± 6.78	86.35 ± 7.05	90.03 ± 5.5
13	Small Brassica chinensis	22507	47.65 ± 8.07	58.29 ± 15.22	48.91 ± 14.62	68.92 ± 5.0	73.37 ± 10.69	73.92 ± 6.73
14	Lactuca sativa	7356	80.13 ± 8.12	92.75 ± 2.24	79.92 ± 6.43	84.7 ± 7.08	89.56 ± 4.7	95.09 ± 1.74
15	Celtuce	1002	94.23 ± 3.21	99.76 ± 0.54	99.58 ± 0.46	92.5 ± 3.43	98.54 ± 2.39	94.45 ± 4.58
16	Film covered lettuce	7262	83.17 ± 3.36	92.72 ± 7.46	90.32 ± 4.39	92.71 ± 5.77	95.82 ± 1.81	96.56 ± 3.01
17	Romaine lettuce	3010	90.15 ± 1.91	97.71 ± 4.09	96.78 ± 1.71	91.46 ± 2.47	95.69 ± 2.81	97.67 ± 2.21
18	Carrot	3217	90.79 ± 4.87	99.57 ± 0.55	96.1 ± 2.61	94.34 ± 4.99	97.34 ± 1.42	98.02 ± 2.38
19	White radish	8712	87.61 ± 3.06	90.18 ± 7.44	90.36 ± 3.7	83.03 ± 2.98	85.06 ± 5.17	79.76 ± 6.96
20	Garlic sprout	3486	85.52 ± 4.37	98.21 ± 1.32	94.31 ± 3.17	94.48 ± 5.87	96.02 ± 4.33	99.76 ± 0.29
21	Broad bean	1328	87.36 ± 6.89	99.95 ± 0.1	98.77 ± 1.0	99.86 ± 0.1	99.65 ± 0.69	100.0 ± 0
22	Tree	4040	87.76 ± 2.0	98.05 ± 2.03	93.45 ± 4.8	97.12 ± 3.07	96.06 ± 7.3	100.0 ± 0
	OA(%)		47.85 ± 4.19	46.21 ± 1.59	43.25 ± 2.78	80.28 ± 2.0	90.61 ± 0.84	88.13 ± 1.08
	AA(%)		75.96 ± 1.33	84.19 ± 1.04	78.59 ± 2.64	86.88 ± 0.79	90.64 ± 1.26	90.77 ± 1.54
	Kappa		45.08 ± 3.55	44.48 ± 1.61	41.5 ± 2.75	76.31 ± 2.17	88.26 ± 1.01	85.36 ± 1.19
	FLOPs(G)		0.015	9.370	8.295	17.17	52.259	31.87
	Param(M)		0.146	0.510	0.471	0.049	0.362	0.071
Overall Accuracy in 5-CV
1	49.22	43.95	43.89	80.56	89.83	89.04
2	46.67	45.19	47.18	82.76	89.96	88.25
3	54.4	47.26	40.31	81.37	91.86	88.95
4	45.29	47.77	44.01	77.62	91.0	86.36
5	43.68	46.86	40.84	79.07	90.4	88.06
Average Accuracy in 5-CV
1	76.9	82.64	78.81	87.4	90.98	91.59
2	75.79	84.66	82.2	87.37	91.52	90.7
3	77.63	84.63	77.44	87.02	91.66	88.13
4	75.21	85.32	79.47	85.5	90.5	91.71
5	74.3	83.71	75.02	87.1	88.54	91.73

Based on Figure 17, it’s observed that MSDesGATnet can also avoid misclassification between different categories of the same crop, such as C7 and C9. Overall, the classification accuracies of most categories exceed 90%. Figure 18 displays the classification maps of all methods on the HongHu dataset, revealing that AMGCFN and the proposed MSDesGATnet have fewer misclassifications and similar classification outcomes. However, since AMGCFN also incorporates the CNN method, its classification results exhibit more noise compared to MSDesGATnet. MSDesGATnet not only ensures accuracy but also provides smoother and more robust classification maps.

Figure 17. Normalized confusion matrix of MSDesGATnet on: (a) PU (b) HongHu.

Figure 18. Classification maps of all methods on HongHu dataset. (a) ground truth (b) DCNN (OA = 43.7%) (c) DBDAnet (OA = 43.9%) (d) SSRN (OA = 40.3%) (e) EGNN (OA = 77.6%) (f) AMGCFN (OA = 90.6%) (g) MSDesGATnet (OA = 89.4%).

Overall, MSDesGAT outperforms other state-of-the-art methods in classification accuracy and performance on multi-scale feature datasets such as PU and spatially continuous single-scale datasets such as IP. For the problem faced by GNN, which is the impure superpixel nodes, MSDesGAT uses a multi-scale connected graph space. Compared to CNN-enhanced methods such as AMGCFN, the classification results are more in line with practical scenarios. At the same time, MSDesGAT introduces an exponential multi-scale feature fusion module to improve classification accuracy, and the overall network has a small number of parameters(M). FLOPs(G) are on the same order of magnitude as other SOTA methods, such as AMGCFN.

Ablation study

The proposed MSDesGATnet encompasses three crucial operations: Multi-scale dense connection, dynamic graph attention mechanism, and multi-scale feature fusion submodule. To demonstrate the effect of the dense connection, we substitute the dense connection with a simple sequential operation and name it MSGATnet. To illustrate that the GNN method extracts features more effectively than the CNN method, we replace GNN with a spatial-spectral convolution operation based on MSGATnet and simulate the scale reduction of MSGATnet with maxpooling, which is named Multi-scale CNN. Single-scale GATnet is designed to highlight the advantages of multi-scales, and MDesGATnet– further removes the SSConv layer for multi-scale feature map fusion.

In this section, the training set partitioning method and cross-validation are kept consistent with the comparison experiments, including the selected samples, and the results on three datasets are demonstrated in Tables 7–10. Among all the results, MSDesGATnet exhibits the best performance and the dynamic graph attention mechanism demonstrates that it is more suitable for small samples than CNN. Furthermore, it’s observed that the use of the SSConv layer can enhance accuracy by approximately 0.4% to 8%. Notably, in datasets containing multiple densely distributed classes (e.g., PU), our proposed densely connected, multi-scale and feature fusion approach can cope with the superpixel impurity problem better than using one of them singularly with an improvement of about 10% in the accuracy precision. In other datasets (e.g., IP, KSC, HongHu), our proposed method also improves the accuracy by 1% to 5% over the single use of densely connected or simply multi-scale. In summary, our proposed multi-scale dense connection, dynamic graph attention mechanism and multi-layer feature fusion prove to be effective.

Table 7. Ablation Study Results on IP. The bolded value in each row indicates the best performance.

Method	MSGATnet	Multi-scale CNN	Single-scale GATnet	MSDesGATnet-	MSDesGATnet
OA(%)	89.68 ± 2.71	68.61 ± 3.35	90.24 ± 0.65	91.06 ± 1.21	91.42 ± 1.9
AA(%)	94.18 ± 1.97	82.54 ± 1.49	93.42 ± 0.65	94.76 ± 0.5	95.63 ± 0.7
Kappa	88.27 ± 3.08	65.62 ± 3.5	89.33 ± 0.73	89.86 ± 1.34	90.28 ± 2.13
FLOPs(G)	0.567	4.413	1.304	0.819	1.304
Param(M)	0.028	1.208	0.062	0.041	0.062

Table 8. Ablation Study Results on KSC. The bolded value in each row indicates the best performance.

Method	MSGATnet	Multi-scale CNN	Single-scale GATnet	MSDesGATnet-	MSDesGATnet
OA(%)	98.88 ± 0.31	92.09 ± 1.28	98.67 ± 1.0	98.49 ± 0.24	98.96 ± 0.38
AA(%)	98.72 ± 0.22	92.42 ± 1.0	98.51 ± 1.27	98.21 ± 0.34	98.82 ± 0.3
Kappa	98.75 ± 0.35	91.21 ± 1.41	98.52 ± 1.11	98.32 ± 0.27	98.84 ± 0.42
FLOPs(G)	8.966	66.901	20.727	11.20	20.727
Param(M)	0.029	1.209	0.066	0.038	0.066

Table 9. Ablation Study Results on PU. The bolded value in each row indicates the best performance.

Method	MSGATnet	Multi-scale CNN	Single-scale GATnet	MSDesGATnet-	MSDesGATnet
OA(%)	72.13 ± 11.13	70.2 ± 1.5	77.69 ± 13.44	80.34 ± 3.84	88.47 ± 1.16
AA(%)	79.28 ± 11.86	73.59 ± 1.4	87.78 ± 4.22	83.39 ± 1.35	88.5 ± 1.29
Kappa	76.57 ± 10.64	62.99 ± 1.69	72.98 ± 14.18	75.13 ± 4.46	84.96 ± 1.42
FLOPs(G)	0.065	31.858	0.113	0.1776	0.113
Param(M)	0.0003	1.150	0.0005	0.001	0.0005

Table 10. Ablation Study Results on HongHu. The bolded value in each row indicates the best performance.

Method	MSGATnet	Multi-scale CNN	Single-scale GATnet	MSDesGATnet-	MSDesGATnet
OA(%)	83.84 ± 2.53	14.97 ± 1.14	87.49 ± 0.53	87.82 ± 1.09	88.13 ± 1.08
AA(%)	86.59 ± 0.52	41.25 ± 3.39	89.52 ± 0.48	89.0 ± 0.88	90.77 ± 1.54
Kappa	80.15 ± 2.87	12.34 ± 0.94	85.19 ± 0.6	85.33 ± 1.34	85.36 ± 1.19
FLOPs(G)	13.82	96.692	32.41	14.942	31.878
Param(M)	0.032	1.213	0.073	0.035	0.071

Multi-scale sensitivity analysis

The value and number of scales within the multi-scale often influence the final classification accuracy. we experimented with the classification accuracy at different scales for three datasets based on the proposed MSDesGATnet. Here, $\mathit{\text{len}}$ represents the number of scales and $k$ denotes the difference in the number of superpixels between scales. For different $k$ and $\mathit{\text{len}}$ values within the same dataset, they all commence from the smallest scale in Table 2.

Figure 19 illustrates the trend of classification accuracy with varying $k$ and $\mathit{\text{len}}$ values. It’s observed that an increase in the scale number doesn’t necessarily lead to a rise in classification accuracy, but there are one or two higher values. For the IP, PU and HongHu datasets, there exists a specific $k$ and $\mathit{\text{len}}$ that render the classification accuracy locally optimal. Meanwhile, for the KSC datasets, the accuracy tends to be higher at smaller scale $\mathit{\text{len}}$ and $k$. Overall, the accuracy gap fluctuates more with different datasets.

Figure 19. Average accuracy at different multiple scales at (a) IP (b) KSC (c) PU (d) HongHu.

Running time

For MSDesGATnet, Table 11 showcases the average running time of all compared methods, the hyperparameters were set identically to the comparison experiment.

Table 11. Running time (seconds) of MSDesGATnet in IP PU KSC dataset.

	DCNN	DBDAnet	SSRN	EGNN	AMGCFN	MSDesGAT
IP	9.26 ± 0.88	141.32 ± 0.1	78.28 ± 0.11	1.7 ± 0.05	13.02 ± 0.94	73.38 ± 1.15
KSC	11.07 ± 0.16	107.06 ± 0.05	59.58 ± 0.07	8.38 ± 0.06	151.39 ± 0.51	1882.86 ± 1.17
HongHu	32.93 ± 0.57	930.58 ± 1.56	256.72 ± 0.43	13.36 ± 0.28	247.64 ± 2.53	1339.7 ± 21.05
PU	12.63 ± 0.45	69.33 ± 0.53	109.24 ± 0.67	8.35 ± 0.4	106.12 ± 1.48	51.83 ± 6.71

Although the testing run time of MSDesGAT on the KSC and HongHu datasets is longer than other methods because KSC and HongHu have more channels after SELF, the running time of MSDesGAT is smaller on other datasets such as IP and PU. To further reduce the running time of the proposed MSDesGAT, a specific number of dimensionality reduction dimensions can be specified, but to some extent, it is not conducive to classification accuracy. In summary, based on the comprehensive classification accuracy, results, and running time, it can be concluded that our proposed MSDesGAT is effective.

Conclusion

In this study, we introduce a novel end-to-end multi-scale graph network structure, MSDesGATnet, for hyperspectral classification. The innovation of MSDesGATnet resides in the fusion of graph attention mechanisms and multi-scale dense networks, with networks at different scales being interconnected and densely connected. The proposed method has demonstrated superior accuracy compared to numerous state-of-the-art methods. In future research, we aim to incorporate multi-scale superpixel segmentation into the end-to-end network. Moreover, it’s feasible to select suitable graph spaces $\mathcal{G}$ based on the resolution of ground features, perform feature enhancement on the corresponding outputs $\mathbf{X}(\mathcal{G}|\cdots )$, or strengthen the weights in the final feature fusion process SSConv to further enhance classification accuracy. Most importantly, our framework enables fellow researchers to utilize our framework when exploring graph node feature fusion algorithms with new datasets, eliminating the need for additional time-consuming searches for original datasets with compatible resolutions.

Additional information

Funding

This work was supported by the National Natural Science Foundation of China (61901471, 61421001), the Project of Construction and Support for high-level Innovative Teams of Beijing Municipal Institutions (BPHR20220123), College Students’ Innovative Entrepreneurial Training Plan Program of China (202211232031) and Beijing Natural Science Foundation (4214072).