How to call setModelData with new value from editor if old value is not valid.

I would say, don’t allow to return from you editor until the user has given a valid input. That’s not the job of the model to check that.