Vehicle classification data are used in many transportation applications, including: pavement design, environmental impact studies, traffic control, and traffic safety (FHWA, 2001). There are several classification methods, including: axle-based (e.g., pneumatic tube and piezoelectric detectors), vehicle length-based (e.g., dual loop and some wayside microwave detectors), as well as emerging machine vision based detection. Each sensor technology has its own strengths and weaknesses regarding costs, performance, and ease of use. Operating agencies spend millions of dollars to deploy vehicle classification stations to collect classified count data, yet very few of these stations are ever subjected to a rigorous performance evaluation to ensure that they are reporting accurate data. As noted in the Traffic Monitoring Guide (FHWA, 2001), the quality of data collected depends on the operating agency to periodically calibrate, test, and validate the performance of classification sensors, but few operating agencies have an on-going performance monitoring system to ensure that well tuned classification stations do not drift out of tune. Both one time and periodic performance monitoring have been prohibitively labor intensive because the only option has been to manually validate the performance, e.g., classifying a sample by hand. When these studies are conducted, the manual classifications are usually of limited value both because the manual data are prone to human error, and among the few studies that have been published, most employ the conventional reporting periods used by the stations (typically 15 min periods or longer), which are too coarse, allowing over-counting errors to cancel under-counting errors. In the present study we develop a classification performance monitoring system to allow operating agencies to rapidly assess the performance of existing classification stations on a per vehicle basis. We eliminate most of the labor demands and instead, deploy a portable non-intrusive vehicle classification system (PNVCS) to classify vehicles, concurrent with an existing classification station. For this study we use a side-fire LIDAR (light detection and ranging) based classifier from Lee and Coifman (2012a) for the PNVCS. Fig. 1 shows a flowchart of our performance evaluation system. The existing classification station normally follows the three boxes within the dashed region (top left of the figure) when it is not under evaluation. The PNVCS is shown immediately to the right of the dashed region. To prevent classification errors from canceling one another in aggregate, we record per-vehicle record (pvr) data in the field from both systems. After the field collection the classification results are evaluated on a per-vehicle basis. Algorithms for time synchronization and for matching observations of a given vehicle between the two classification systems are developed in this study. These algorithms automatically compare the vehicle classification between the existing classification station and the PNVCS for each vehicle. If the two systems agree, the given vehicle is automatically taken as a success by the classification station under the implicit assumptions: (i) That few vehicles will be misclassified the same way by the two independent classification systems. (ii) That the PNVCS has sufficient accuracy so that its data can be used as a benchmark for the existing classification station (in this case Lee and Coifman, 2012a, found that the LIDAR system classified vehicles with 99.5% accuracy on an evaluation set of 21,769 non-occluded vehicles). The temporary deployment includes a video camera mounted close to the LIDAR sensors and pointed at their detection zone (right-most path in Fig. 1) to allow a human to assess any discrepancies. A human only looks at a given vehicle when the two systems disagree, and for this task we have developed tools to semi-automate the manual validation process, greatly increasing the efficiency and accuracy of the human user. The data sets in this study take only a few minutes for the user to validate an hour of pvr data from a multi-lane facility.