Abstract
This paper presents RPEP: RGB Pre-training for Event-based hand Pose estimation, the first framework that uses labeled RGB images and unpaired, unlabeled event data to learn event-based estimators. Event data offers significant benefits such as high temporal resolution and low latency, but its application to hand pose estimation is still limited by the scarcity of labeled training data. To address this, we repurpose real RGB datasets to train event-based estimators by constructing pseudo-event-RGB pairs, i.e., generated event data aligned with the ground-truth poses of RGB images. However, existing construction methods struggle to produce realistic events for articulated hand motion. The core issue is a temporal-resolution mismatch: RGB data is usually captured at low frame rates, while events are generated at much higher temporal resolution, causing inter-frame information loss. To resolve the temporal-resolution mismatch, RPEP densifies inter-frame hand motion for pseudo-event construction. We decompose each hand motion into step-by-step movements, recovering missing inter-frame dynamics and producing pseudo-events that are more faithful to real event streams. Additionally, RPEP imposes a motion reversal constraint, regularizing event generation using reversed motion. Extensive experiments show that our pre-trained model significantly outperforms state-of-the-art methods on real event data, achieving up to 24% improvement on EvRealHands. Moreover, it delivers strong performance with minimal labeled samples for fine-tuning, making it well-suited for practical deployment.