Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-image Generation
Date
Authors
Li, Ruijun
Li, Weihua
Yang, Yi
Wei, Hanyu
Jiang, Jianhua
Bai, Quan
Supervisor
Item type
Journal Article
Degree name
Journal Title
Journal ISSN
Volume Title
Publisher
Springer Science and Business Media LLC
Abstract
Recently, diffusion models have been proven to perform remarkably well in text-to-image synthesis tasks in a number of studies, immediately presenting new study opportunities for image generation. Google’s Imagen follows this research trend and outperforms DALLE2 as the best model for text-to-image generation. However, Imagen merely uses a T5 language model for text processing, which cannot ensure learning the semantic information of the text. Furthermore, the Efficient UNet leveraged by Imagen is not the best choice in image processing. To address these issues, we propose the Swinv2-Imagen, a novel text-to-image diffusion model based on a Hierarchical Visual Transformer and a Scene Graph incorporating a semantic layout. In the proposed model, the feature vectors of entities and relationships are extracted and involved in the diffusion model, effectively improving the quality of generated images. On top of that, we also introduce a Swin-Transformer-based UNet architecture, called Swinv2-Unet, which can address the problems stemming from the CNN convolution operations. Extensive experiments are conducted to evaluate the performance of the proposed model by using three real-world datasets, i.e. MSCOCO, CUB and MM-CelebA-HQ. The experimental results show that the proposed Swinv2-Imagen model outperforms several popular state-of-the-art methods.Description
Keywords
4605 Data Management and Data Science, 46 Information and Computing Sciences, 4603 Computer Vision and Multimedia Computation, 0801 Artificial Intelligence and Image Processing, 0906 Electrical and Electronic Engineering, 1702 Cognitive Sciences, Artificial Intelligence & Image Processing, 4602 Artificial intelligence, 4611 Machine learning, Text-to-image synthesis, Diffusion models, Scene graph, Graph neural network, UNet
Source
Neural Computing and Applications, ISSN: 0941-0643 (Print); 1433-3058 (Online), Springer Science and Business Media LLC, 36(28), 17245-17260. doi: 10.1007/s00521-023-09021-x
Publisher's version
Rights statement
Open Access. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.