1372798 - A hybrid deep learning paradigm for image-based vehicle classification

Vehicle classification data is fundamental to transportation infrastructure planning, design and operations. This type of data has been traditionally collected by permanent count and weight-in-motion stations, which are expensive to install and maintain. The quality of data largely depends on the types of sensors, which often require labor-intensive field calibration. Given the existing deployment of video-based traffic monitoring systems at the network level by state departments of transportation, vehicle classification data can be conveniently acquired through such systems by artificial intelligence-based methods. In this study, a hybrid deep learning architecture is introduced for vehicle classification by fusing features obtained from supervised and self-supervised learning methods. Supervised learning is employed to finetune a pretrained YOLOR object detector for detecting vehicle wheels, from which definitive wheel positional features are retrieved. Self-supervised learning is used to extract explicit image features (e.g., semantic segmentation) from abundant unlabeled images. To improve the robustness of image features and facilitate their complementarity with the wheel positional features for the classification task, a wheel masking strategy is implemented while the pretrained backbone network is further finetuned. Our experiments show that finetuning vision transformer models pretrained by self-supervised learning methods, such as DINO (self-distillation with no labels) and data2vec, leads to better classification results. Particularly, use of the data2vec-derived representations, which are consistent with our wheel masking strategy, results in a celebrated Top-1 classification accuracy of 97.2% for classifying 13 vehicle categories defined by Federal Highway Administration.