An on-board autonomous visual tracking system and method for autonomously tracking and simultaneously providing surveillance of a ground stationary target (GST) or ground moving target (GMT) from unmanned air vehicle (UAV). In the embodiment, the on-board gimbaled camera takes the real-time video frame sequence and sends to the on-board autonomous visual tracking system. The embodiment on-board autonomous visual tracking system further comprises a multi-level autonomous visual tracking algorithm to track the target continuously. The embodiment on-board autonomous visual tracking system further comprises and provides the target pixel position to compute the position of target in the real-world coordinate frame. In the embodiment the EKF has been used as target state estimator which provides current position and velocity of the target based on target pixel position and unmanned aerial vehicle position, velocity, attitude and camera attitude which is provided by INS/GPS sensor and gimbaled camera. The non-linear unmanned aerial vehicle guidance law, which provides unmanned aerial vehicle heading rate and velocity rate, while the camera’s pan rate and tilt rate is being computed by camera control law for continuous tracking of the target. The embodiment method further comprises and tracks the target even target moves quickly, similar color of another target, and similar background color of target.