Building Change Detection (BCD) holds significant value in the context of monitoring land use, while Building Damage Assessment (BDA) plays a crucial role in expediting humanitarian rescue efforts post-disasters. To address these needs, we propose the Bi-Temporal Attention Module (BAM) as an innovative cross-attention mechanism aimed at effectively capturing spatio-temporal semantic relations between a pair of bi-temporal remote sensing images. Within BAM, a shifted windowing scheme has been implemented to confine the scope of the cross-attention mechanism to a specific range, not only excluding remote and irrelevant information but also contributing to computational efficiency. Moreover, existing methods for BDA often overlook the inherent order of ordinal labels, treating the BDA task simplistically as a multi-class semantic segmentation problem. Recognizing the vital significance of ordinal relationships, we approach the BDA task as an ordinal regression problem. To address this, we introduce a rank-consistent ordinal regression loss function to train our proposed change detection network, Bi-temporal Attention Transformer (BAT). Our method achieves state-of-the-art accuracy on two BCD datasets (LEVIR-CD+ and S2Looking), as well as the largest BDA dataset (xBD).