GameObject jump to other clients position at drag

Hi,
I am playing around with UNET and i spawn a "Orange" GameObject, with the correct Collider2D etc.

I have two Clients, #1 and #2. Each client have a GameObject,GO1 and GO2.

After spawning i drag one of the GO's away from the other, all works. If i then, as #1, click and drag from GO2 (#2) the GO1 jumps to that position and drags from there. This is the problem as i want the GO1 only to be dragged when i click and drag on that object.

Here is the code i use, script attached to...

GameObject jump to other clients position at drag