A method and system relate to operating an ADV. When an ADV transitions from a manual driving mode to an autonomous driving mode, a first speed reference is determined based on a current position of the ADV. The current position of the ADV is dynamically measured in response to a speed control command issued in a previous command cycle and a target speed of a current command cycle. A second speed reference is determined based on a current target position for a current command cycle. A speed control command is then generated for controlling the speed of the ADV in the autonomous driving mode based on the first speed reference, the second speed reference, and the target speed of the ADV for the current command cycle, such that the ADV operates in a similar acceleration rate or deceleration rate before and after transitioning from the manual driving mode to the autonomous driving mode.