Tags Re-ranking Using Multi-level Features in Automatic Image Annotation

Subject Areas : Image, Speech and Signal Processing

Forogh Ahmadi ¹ , Vafa Maihami ²

1 - Department of Computer Engineering, Islamic Azad University, Sanandaj Branch, Sanandaj, Iran
2 - Department of Computer Engineering, Islamic Azad University, Sanandaj Branch, Sanandaj, Iran

Received: 2019-07-06 Accepted : 2019-08-22 Published : 2019-11-01

Keywords: Tag ranking, Neighbor voting, Low level feature, Automatic image annotation,

Abstract :

Automatic image annotation is a process in which computer systems automatically assign the textual tags related with visual content to a query image. In most cases, inappropriate tags generated by the users as well as the images without any tags among the challenges available in this field have a negative effect on the query's result. In this paper, a new method is presented for automatic image annotation with the aim at improving the obtained tags, as well as reducing the effect of unrelated tags. In the proposed method, first, the initial tags are determined by extracting the low-level features of the image and using neighbor voting method. Afterwards, each initial tag is assigned by a degree based on the neighbor image features of the query image. Finally, they will be ranked based on summing the degrees of each tag and the best tags will be selected by removing the unrelated tags. The experiments conducted on the proposed method using the NUS-WIDE dataset and the commonly used evaluation metrics demonstrate the effectiveness of the proposed system compared to the previous works.

References:

[1] R. Datta, D. Joshi, J. Li, and J. Z. Wang, “Image retrieval: Ideas, influences, and trends of the new age,” ACM Comput. Surv., vol. 40, no. 2, pp. 1–60, Apr. 2008.
[2] X. Chang, H. Shen, S. Wang, J. Liu, and X. Li, “Semi-supervised Feature Analysis for Multimedia Annotation by Mining Label Correlation,” pp. 74–85, 2014.
[3] X. Li, T. Uricchio, L. Ballan, M. Bertini, C. G. M. Snoek, and A. Del Bimbo, “Socializing the semantic gap: A comparative survey on image tag assignment, refinement, and retrieval,” ACM Comput. Surv., vol. 49, no. 1, pp. 1–39, Jun. 2016.
[4] M. Wang, B. Ni, X.-S. Hua, and T.-S. Chua, “A survey of multimedia tagging with human-computer joint exploration.,” ACM Comput. Surv., vol. 44, no. 4, pp. 1–24, Aug. 2012.
[5] Yue Gao, Meng Wang, Zheng-Jun Zha, Jialie Shen, Xuelong Li, and Xindong Wu, “Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search,” IEEE Trans. Image Process., vol. 22, no. 1, pp. 363–376, Jan. 2013.
[6] Y. Liu, D. Zhang, G. Lu, and W.-Y. Ma, “A survey of content-based image retrieval with high-level semantics,” Pattern Recognit., vol. 40, no. 1, pp. 262–282, Jan. 2007.
[7] D. Zhang, M. M. Islam, and G. Lu, “A review on automatic image annotation techniques,” Pattern Recognit., vol. 45, no. 1, pp. 346–362, Jan. 2012.
[8] A. E. Brito, D. Kletter, M. Singhal, and M. Bern, “Benchmark study of automatic annotation of MALDI-TOF N-glycan profiles,” J. Proteomics, vol. 129, pp. 71–77, Nov. 2015.
[9] S. Protasov, A. M. Khan, K. Sozykin, and M. Ahmad, “Using deep features for video scene detection and annotation,” Signal, Image Video Process., pp. 1–9, Jan. 2018.
[10] X. Xirong Li, C. G. M. Snoek, and M. Worring, “Learning Social Tag Relevance by Neighbor Voting,” IEEE Trans. Multimed., vol. 11, no. 7, pp. 1310–1322, Nov. 2009.
[11] D. Tian and Z. Shi, “Automatic image annotation based on Gaussian mixture model considering cross-modal correlations,” J. Vis. Commun. Image Represent., vol. 44, pp. 50–60, Apr. 2017.
[12] K. Akhilesh and R. R. Sedamkar, “Automatic image annotation using an ant colony optimization algorithm (ACO),” in 2016 IEEE 7th Power India International Conference (PIICON), 2016, pp. 1–4.
[13] Q. Cheng, Q. Zhang, P. Fu, C. Tu, and S. Li, “A survey and analysis on automatic image annotation,” Pattern Recognit., vol. 79, pp. 242–259, Jul. 2018.
[14] S. Lee, W. De Neve, and Y. M. Ro, “Visually weighted neighbor voting for image tag relevance learning,” Multimed. Tools Appl., vol. 72, no. 2, pp. 1363–1386, Apr. 2013.
[15] G. Carneiro, A. B. Chan, P. J. Moreno, and N. Vasconcelos, “Supervised Learning of Semantic Classes for Image Annotation and Retrieval,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 394–410, Mar. 2007.
[16] T. Uricchio, L. Ballan, M. Bertini, and A. Del Bimbo, “An evaluation of nearest-neighbor methods for tag refinement,” in 2013 IEEE International Conference on Multimedia and Expo (ICME), 2013, pp. 1–6.
[17] T. Uricchio, L. Ballan, L. Seidenari, and A. Del Bimbo, “Automatic image annotation via label transfer in the semantic space,” Pattern Recognit., vol. 71, pp. 144–157, Nov. 2017.
[18] X. Zhu, W. Nejdl, and M. Georgescu, “An adaptive teleportation random walk model for learning social tag relevance,” in Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval - SIGIR ’14, 2014, pp. 223–232.
[19] Z. Li, J. Liu, C. Xu, and H. Lu, “MLRank: Multi-correlation Learning to Rank for image annotation,” Pattern Recognit., vol. 46, no. 10, pp. 2700–2710, Oct. 2013.
[20] J. Johnson, L. Ballan, and L. Fei-Fei, “Love Thy Neighbors: Image Annotation by Exploiting Image Metadata,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4624–4632.
[21] C. Cui, J. Shen, J. Ma, and T. Lian, “Social tag relevance learning via ranking-oriented neighbor voting,” Multimed. Tools Appl., vol. 76, no. 6, pp. 8831–8857, Mar. 2017.
[22] D. Liu, X.-S. Hua, L. Yang, M. Wang, and H.-J. Zhang, “Tag ranking,” in Proceedings of the 18th international conference on World wide web - WWW ’09, 2009, p. 351.
[23] Y. Wang, X. Lin, L. Wu, and W. Zhang, “Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval,” IEEE Trans. Image Process., vol. 26, no. 3, pp. 1393–1404, Mar. 2017.
[24] L. Ballan, M. Bertini, G. Serra, and A. Del Bimbo, “A data-driven approach for tag refinement and localization in web videos,” Comput. Vis. Image Underst., vol. 140, pp. 58–67, Nov. 2015.
[25] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “NUS-WIDE: a real-world web image database from National University of Singapore,” in Proceeding of the ACM International Conference on Image and Video Retrieval - CIVR ’09, 2009.
[26] F. Tian, X. Shen, and X. Liu, “Multimedia automatic annotation by mining label set correlation,” Multimed. Tools Appl., vol. 77, no. 3, pp. 3473–3491, Feb. 2018.
[27] Maihami V, Yaghmaee F. Automatic image annotation using community detection in neighbor images. Physica A: Statistical Mechanics and its Applications. 1;507:123-32, 2018.
[28] Maihami V, Yaghmaee F. A genetic-based prototyping for automatic image annotation. Computers & Electrical Engineering. 1;70:400-12, 2018.
[29] Lotfi A, Maihami V, Yaghmaee F. Wood image annotation using gabor texture feature. Int. J. Mechatronics, Electr. Comput. Technol..;4:1508-23, 2014.
[30] Maihami V, Yaghmaee F. A review on the application of structured sparse representation at image annotation. Artificial Intelligence Review. 1;48(3):331-48, 2017.