DNS for domains on multiple servers

[Jimpoo]

I currently have a Server A, and I offer my client two name servers, let's say:

NS1.MyDNSDomain.com
NS2.MyDNSDomain.com

I have no problems for hosting websites on Server A.

Now I have a new Server B. In order to offer my client the same name servers as Server A, and the websites will be hosted on Server B, how should I setup the DNS on Server B and/or Server A?

Do I have to create two new name servers like following?
NS3.MyDNSDomain.com
NS4.MyDNSDomain.com

This should be a very common issue for many hosting companies, but I'm new to this, anybody help?

Thank you!