Diffusion Model for A Virtual Try-On System

Abstract

We present a modular virtual try-on (VTON) system that integrates natural language control, efficient diffusion-based image synthesis, and lightweight garment classification. User intent is parsed by a large language model (LLM) into structured visual prompts. A LoRA-tuned diffusion model generates tryon images conditioned on pose and segmentation maps, while a compact classifier, LightClothNet, handles five-category clothing recognition and pre-filtering. The pipeline is built using ComfyUI nodes and orchestrated via Dify. Compared to the existing methods, the proposed system offers improved realism, garment-pose alignment, and controllability. Our evaluations on the DressCode and VITON-HD datasets show that LoRA fine-tuning enhances fidelity under limited data, while LightClothNet achieves up to 91.76% precision and 0.91 F1-score with low latency. This result demonstrates how multimodal control, lightweight classification, and diffusion generation are unified for fast, flexible, and userdriven VTON applications.