Why does RSA decryption need to verify the public key?

As far as I know, N + E is the public key, and N + D is the private key.
So I don’t know why the public key E needs to be verified when decrypting with a private key.
In theory, E is not necessary.