Enhancing Thai food recognition through multimodal fusion of image and Fourier spectrum

Abstract

Recognizing food is a challenging task in artificial intelligence research because food items may undergo deformations during cooking or serving, and they can be partially or fully occluded, making it difficult for recognition systems to analyze their complete visual information. Therefore, in addition to evaluating object detection effectiveness, consideration must be given to the texture of the food. However, Convolutional Neural Networks may fall short in capturing textural information. In this research, we propose a method to enhance the efficiency of Thai food recognition by employing the concept of multi-modal fusion, incorporating Fourier Spectrum images to take texture representation into account and improve the model’s performance. In the fusion process, we employed the CentralNet framework and compared it with baselines (using only images and conventional concatenation fusion) on three datasets: THFOOD-50, FoodyDudy, and our newly proposed food dataset called iTHFOOD-200. This new dataset encompasses a more diverse range of food types. The experimental results demonstrate that fusion with the CentralNet framework yielded better performance than the baselines using only images (7.1% Top-1 Accuracy and 3.9% Top-5 Accuracy on average) and conventional concatenation fusion (5.0% Top-1 Accuracy and 2.4% Top-5 Accuracy on average).