Yes the D800 would automatically enter DX mode with a DX lens attached (unless it’s set not to —auto dx crop set to off in shooting menu)

The vignetting caused by a DX lens is pretty unmistakeable though, are we sure that’s HansPeter’s problem?