Gaussian Process Assisted Labeling for Video Object Tracking: An Analysis
Resumo
The development of datasets for video object tracking is inherently resource-intensive, primarily due to the requirement of exhaustive frame-by-frame annotation of target positions. Various strategies have been explored to mitigate these costs. This study evaluates the effectiveness of Gaussian Process (GP) Regression as an auxiliary tool for annotating target trajectories within video tracking benchmarks. By leveraging manual ground-truth data sampled at intervals of k frames, GP models are employed to infer the coordinates for the intervening unlabeled frames. Experimental evaluations conducted on Multi-Object Tracking (MOT) datasets demonstrate a mean Area Under the Curve (AUC) exceeding 90% at k = 20 for sequences characterized by uniform target dynamics and stationary camera platforms. Furthermore, the proposed approach sustained a mean precision higher than 80% at same k even when subjected to more challenging and unconstrained scenarios.
Referências
Castro-Girón, E. and Shiguihara, P. (2021). A method for dataset labeling for activity recognition in videos. In 2021 IEEE XXVIII International Conference on Electronics, Electrical Engineering and Computing (INTERCON), pages 1–4.
Cetintas, O., Brasó, G., and Leal-Taixé, L. (2023). Unifying short and long-term tracking with graph hierarchies. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22877–22887.
Chang, N., Ferroni, F., Tarr, M. J., Hebert, M., and Ramanan, D. (2023). Thinking like an annotator: Generation of dataset labeling instructions.
Cioppa, A., Giancola, S., Deliège, A., Kang, L., Zhou, X., Cheng, Z., Ghanem, B., and Van Droogenbroeck, M. (2022). Soccernet-tracking: Multiple object tracking dataset and benchmark in soccer videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3491–3502.
Cui, Y., Zeng, C., Zhao, X., Yang, Y., Wu, G., and Wang, L. (2023). Sportsmot: A large multi-object tracking dataset in multiple sports scenes. arXiv preprint arXiv:2304.05170.
Dai, K., Zhao, J., Wang, L., Wang, D., Li, J., Lu, H., Qian, X., and Yang, X. (2021). Video annotation for visual tracking via selection and refinement.
Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taixé, L. (2020). Mot20: A benchmark for multi object tracking in crowded scenes.
Gao, R., Qi, J., and Wang, L. (2025). Multiple object tracking as id prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27883–27893.
Gil-Jiménez, P., Gómez-Moreno, H., López-Sastre, R., and Maldonado-Bascón, S. (2016). Geometric bounding box interpolation: an alternative for efficient video annotation. EURASIP Journal on Image and Video Processing, 2016(1):8.
Gutiérrez, J., Gutiérrez, V., Mora, Á., Rodriguez, S., and Blanco, J. L. (2025). An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage. arXiv e-prints, page arXiv:2510.21798.
Horych, T., Mandl, C., Ruas, T., Greiner-Petter, A., Gipp, B., Aizawa, A., and Spinde, T. (2025). The promises and pitfalls of llm annotations in dataset labeling: a case study on media bias detection.
Huang, Q. and Zhao, T. (2024). Data collection and labeling techniques for machine learning.
Krenzer, A., Makowski, K., Hekalo, A., Fitting, D., Troya, J., Zoller, W. G., Hann, A., and Puppe, F. (2022). Fast machine learning annotation in the medical domain: a semi-automated video annotation tool for gastroenterologists. BioMedical Engineering OnLine, 21(1):33.
Maclang, A. R., Orante, M. L., Salvador, R. D., Del Carmen, D. J., and Cajote, R. D. (2023). Video dataset labeling using active learning with applications in vehicle classification and traffic flow rate measurement. In TENCON 2023 - 2023 IEEE Region 10 Conference (TENCON), pages 936–941.
Maggiolino, G., Ahmad, A., Cao, J., and Kitani, K. (2023). Deep oc-sort: Multi-pedestrian tracking by adaptive re-identification. In 2023 IEEE International Conference on Image Processing (ICIP), pages 3025–3029.
Milan, A., Leal-Taixe, L., Reid, I., Roth, S., and Schindler, K. (2016). Mot16: A benchmark for multi-object tracking.
Muller, M., Bibi, A., Giancola, S., Alsubaihi, S., and Ghanem, B. (2018). Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pages 300–317.
Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.
Sun, P., Cao, J., Jiang, Y., Yuan, Z., Bai, S., Kitani, K., and Luo, P. (2022). Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Tarsoly, S., Galambos, P., and Károly, A. I. (2025). Comparison of manual and ai-assisted labeling techniques in pixel-wise instance segmentation. In 2025 IEEE 23rd World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 000115– 000122.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA.
Wojke, N., Bewley, A., and Paulus, D. (2017). Simple online and realtime tracking with a deep association metric. In 2017 IEEE International Conference on Image Processing (ICIP), pages 3645–3649.
Wu, Y., Lim, J., and Yang, M.-H. (2013). Online object tracking: A benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2411–2418.
Yang, F. (2022). A multi-person video dataset annotation method of spatio-temporally actions.
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., and Wang, X. (2022). Bytetrack: Multi-object tracking by associating every detection box. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, page 1–21, Berlin, Heidelberg. Springer-Verlag.
Cetintas, O., Brasó, G., and Leal-Taixé, L. (2023). Unifying short and long-term tracking with graph hierarchies. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22877–22887.
Chang, N., Ferroni, F., Tarr, M. J., Hebert, M., and Ramanan, D. (2023). Thinking like an annotator: Generation of dataset labeling instructions.
Cioppa, A., Giancola, S., Deliège, A., Kang, L., Zhou, X., Cheng, Z., Ghanem, B., and Van Droogenbroeck, M. (2022). Soccernet-tracking: Multiple object tracking dataset and benchmark in soccer videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3491–3502.
Cui, Y., Zeng, C., Zhao, X., Yang, Y., Wu, G., and Wang, L. (2023). Sportsmot: A large multi-object tracking dataset in multiple sports scenes. arXiv preprint arXiv:2304.05170.
Dai, K., Zhao, J., Wang, L., Wang, D., Li, J., Lu, H., Qian, X., and Yang, X. (2021). Video annotation for visual tracking via selection and refinement.
Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taixé, L. (2020). Mot20: A benchmark for multi object tracking in crowded scenes.
Gao, R., Qi, J., and Wang, L. (2025). Multiple object tracking as id prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27883–27893.
Gil-Jiménez, P., Gómez-Moreno, H., López-Sastre, R., and Maldonado-Bascón, S. (2016). Geometric bounding box interpolation: an alternative for efficient video annotation. EURASIP Journal on Image and Video Processing, 2016(1):8.
Gutiérrez, J., Gutiérrez, V., Mora, Á., Rodriguez, S., and Blanco, J. L. (2025). An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage. arXiv e-prints, page arXiv:2510.21798.
Horych, T., Mandl, C., Ruas, T., Greiner-Petter, A., Gipp, B., Aizawa, A., and Spinde, T. (2025). The promises and pitfalls of llm annotations in dataset labeling: a case study on media bias detection.
Huang, Q. and Zhao, T. (2024). Data collection and labeling techniques for machine learning.
Krenzer, A., Makowski, K., Hekalo, A., Fitting, D., Troya, J., Zoller, W. G., Hann, A., and Puppe, F. (2022). Fast machine learning annotation in the medical domain: a semi-automated video annotation tool for gastroenterologists. BioMedical Engineering OnLine, 21(1):33.
Maclang, A. R., Orante, M. L., Salvador, R. D., Del Carmen, D. J., and Cajote, R. D. (2023). Video dataset labeling using active learning with applications in vehicle classification and traffic flow rate measurement. In TENCON 2023 - 2023 IEEE Region 10 Conference (TENCON), pages 936–941.
Maggiolino, G., Ahmad, A., Cao, J., and Kitani, K. (2023). Deep oc-sort: Multi-pedestrian tracking by adaptive re-identification. In 2023 IEEE International Conference on Image Processing (ICIP), pages 3025–3029.
Milan, A., Leal-Taixe, L., Reid, I., Roth, S., and Schindler, K. (2016). Mot16: A benchmark for multi-object tracking.
Muller, M., Bibi, A., Giancola, S., Alsubaihi, S., and Ghanem, B. (2018). Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pages 300–317.
Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.
Sun, P., Cao, J., Jiang, Y., Yuan, Z., Bai, S., Kitani, K., and Luo, P. (2022). Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Tarsoly, S., Galambos, P., and Károly, A. I. (2025). Comparison of manual and ai-assisted labeling techniques in pixel-wise instance segmentation. In 2025 IEEE 23rd World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 000115– 000122.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA.
Wojke, N., Bewley, A., and Paulus, D. (2017). Simple online and realtime tracking with a deep association metric. In 2017 IEEE International Conference on Image Processing (ICIP), pages 3645–3649.
Wu, Y., Lim, J., and Yang, M.-H. (2013). Online object tracking: A benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2411–2418.
Yang, F. (2022). A multi-person video dataset annotation method of spatio-temporally actions.
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., and Wang, X. (2022). Bytetrack: Multi-object tracking by associating every detection box. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, page 1–21, Berlin, Heidelberg. Springer-Verlag.
Publicado
19/07/2026
Como Citar
MAIA, Felipe J. A.; MAIA, José E. B..
Gaussian Process Assisted Labeling for Video Object Tracking: An Analysis. In: SEMINÁRIO INTEGRADO DE SOFTWARE E HARDWARE (SEMISH), 53. , 2026, Gramado/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 358-369.
ISSN 2595-6205.
DOI: https://doi.org/10.5753/semish.2026.23473.
