We ourselves are using this principle of one (well two in fact, as we use three authoritative nameservers) slave server for many (50+ currently) master servers as well, so my 2¢ are:
You can do that easily and out of the box with PowerDNS and it's "Supermaster" feature.
PowerDNS does save/remember from which primary DNS it "learned" a zone from and will then receject updates for this zone from any other nameserver.
So you are completely safe from hijacking and only need to manually interfere (i.e. delete the zone from the secondary nameserver(s)) if you really wan't another/new primary to be authoritative for that domain.
p.s. this feature may only be available if you use a database backend for PowerDNS. (and not legacy bind zonefiles - which are also supported by PowerDNS)