Swaping a point upwards removes the former point on the position.
Swaping a point downwards removes the next point on the position.
To fix, we have to keep the former/next point and write it back. If swap is not possible (former/next index does not exist) the method should do nothing and return false.