Fingerprint sensor becomes more and more popular, and has been installed in many electronic devices, ranging from portable mobile devices to high-end personal computers. With the high penetration power of fingerprint sensor, this paper proposes to make use of finger gesture as a natural physical input device through a sweep fingerprint sensor which is small, compact and cost effective. To the best of our knowledge, this is the first attempt to perform finger gesture recognition on sweep sensor. The contributions of this paper are threefold. First, we define a set of finger motion features for finger gesture recognition. Second, we develop an efficient algorithm to detect the finger motion features. Finally, we propose and develop an efficient algorithm to recognize four finger gestures. Experimental results show that the proposed method could recognize four gestures with 95% accuracy. The computational time to recognize one finger gesture is less than 70 ms in a standard PC. The encouraging results show that the proposed finger gesture recognition algorithm is ready to be deployed to consumer electronics and provides a new mean for natural input.