The acceleration of the body "does not" depend on the mass (for freely falling). Hence both man and ball have same acceleration. Hence their velocities at every point would be same. So "Relative" velocity is 0. Hence the ball will always remains stationary as seen by him