Gravitational potential energy between two masses (M) separated by a distance R is −GMM/R=−GM2/R.
When a small mass m is taken out of mass M, and kept at a distance x from (M−m), the total PE of the system becomes −GMm/(R−x)−Gm(M−m)/x−GM(M−m)/R
As m moves towards M, the gravitational PE in the system will increase. Mass (M-m) will attract mass m and external for will be required to move m away from (M-m).
Once mass m reaches a point P where R1 is the distance between mass m and M and R2 is the distance between m and (M−m) and R1 and R2 related as R1/R2=M/(M−m), the potential energy on mass m will be zero. At this point mass m will not experience any force.
As mass m stars moving towards mass M, the PE will decrease as mass M will attract mass m.
So, in this situation, PE of the total system will increase first and then decrease.