Frequency-aware Spatio-temporal Topology Learning for Skeleton-based Human Activity Recognition

Abstract

Skeleton-based human activity recognition (HAR) has made significant progress through graph convolutional networks (GCNs) and Transformer architectures for spatiotemporal modeling. However, existing methods either employ predefined static graph topologies that cannot adapt to heterogeneous skeleton data or learn dynamic topologies based solely on local spatiotemporal features, thereby overlooking the global temporal frequency features of joint movements that are important for discovering semantically meaningful spatial relationships. We propose Frequency-Aware Topology Learning Graph Convolutional Network (FATL-GCN), a novel architecture that integrates frequency-aware temporal context to guide adaptive learning of spatial topology. Our approach leverages Time-to-Vector linear frequency encoding to capture both periodic and non-periodic motion patterns, employs frequency-guided topology learning to generate action-specific graphs through temporal-context-driven attention, and incorporates hierarchical multi-scale fusion for robust feature extraction across scales. Extensive experiments achieved top-1 accuracies of 93.8% (cross-subject) and 97.5% (cross-view) on NTU-60, 91.9% (cross-subject) and 93.1% (cross-setup) on NTU-120, and 51.7% on Kinetics-Skeleton. Ablation studies confirm the critical role of our components, with removing the dynamic graph topology causing a 3.5% accuracy drop and removing frequency-aware encoding causing a 2.1% drop.