Boosting PyTorch Inference on CPU: From Post-Training Quantization to Multithreading






The BirdCLEF competitions are a series of annually recurring competitions on Kaggle. The main objective of a BirdCLEF competition is usually to identify a specific bird species by sound. The competitors are given short audio files of single bird calls and then must predict whether a specific bird was present in a longer recording.

In an earlier edition of The Kaggle Blueprints, we have already reviewed the winning approaches to audio classification with Deep Learning from last year’s “BirdCLEF 2022” competition.

One aspect that was novel in the “BirdCLEF 2023” competition was the limited time and computational constraints: Competitors were asked to predict roughly 200 10-minute-long recordings on a CPU Notebook within 2 hours.

Now, you might be asking why anyone would want to infer a Deep Learning model on a CPU instead of a GPU. This is a common practical problem statement [4] as oftentimes staff (especially in conservation but also in other industries) have budget constraints and thus only have access to limited computing resources. Additionally, being able to make predictions quickly is helpful.