James is right, the workaround is to make a layer copy, select the lot in the new layer, then go to the canvas size and increase it by a known amount. Then contract your selection, then smooth it, place a new layer below the top one and colour it or whatever you were going to do, flatten the image and the resize you canvas to the original. There are probably other methods.
Attached is a very quick and dirty example with ugly colours to show what I mean. By the way, the PSD was at 16 bit depth, I had to make it 8 bit to be able to make the JPG.