For those having the exact same problem here is the link that helped me sort it out:
https://forums.aws.amazon.com/message.jspa?messageID=404398
In my opinion it is step 2 that made the difference:
2. Configured and added the ifcfg script
# ifconfig eth1 <INTERNAL_IP_1> netmask <YOUR_NETMASK>
# ifconfig eth2 <INTERNAL_IP_2> netmask <YOUR_NETMASK>
In "most(?)" cases the IPs given by AWS are on the same subnet mas so IFCONFIG should be able to give you that. Be sure to replace the "sample IPs", "sample SUBNETs", etc. with those of your instance.
In case that link ever goes down (It's an oldie but still valid) -- a copy/paste of that link:
Posted by:
Yogi@AWS on Nov 29, 2012
Probably this is a issue of default gateway. Here is a step by step approach that you can take to acomplish it. Lets say you have
eth0 - 10.0.0.10
eth1 - 10.0.0.11
eth2 - 10.0.0.12
1. Turned off iptables & Selinux (Just for test, you may enable it later)
# service iptables stop
# chkconfig iptables off
# vi /etc/selinux/config
changed the policy from enforcing to disabled.
2. Configured and added the ifcfg script
# ifconfig eth1 10.0.0.11 netmask 255.255.255.0
# ifconfig eth2 10.0.0.12 netmask 255.255.255.0
# vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
PERSISTENT_DHCLIENT=yes
DEFROUTE=no
EC2SYNC=yes
Similarly for eth2
3. Bring up the interfaces & reboot the machine
# ifup eth1
# ifup eth2
# init 6
4. SSH'd back to the instance and added routes and rules
# ip route show
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.10
10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.11
10.0.0.0/24 dev eth2 proto kernel scope link src 10.0.0.12
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
169.254.0.0/16 dev eth2 scope link metric 1004
default via 10.0.0.1 dev eth0
# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup
default
Add routes --
# ip route add
default via 10.0.0.1 dev eth0 tab 1
# ip route add
default via 10.0.0.1 dev eth1 tab 2
# ip route add
default via 10.0.0.1 dev eth2 tab 3
Make a note, I'm assuming that the ENIs are all in the same subnet. If they are in different subnet, make sure you add routes to the respective gateway for that subnet.
Add rules --
# ip rule add from 10.0.0.10/32 tab 1 priority 500
# ip rule add from 10.0.0.11/32 tab 2 priority 600
# ip rule add from 10.0.0.12/32 tab 3 priority 700
Check the rules. This will now show the metrics
# ip rule show
0: from all lookup local
500: from 10.0.0.10 lookup 1
600: from 10.0.0.11 lookup 2
700: from 10.0.0.12 lookup 3
32766: from all lookup main
32767: from all lookup
default
At this point you will be able to telnet, ssh other ENI using EIP. Do note, add routes & add rules part of this guide in step 4 is not persistent and after every reboot it will be lost. You may create a start-up script to do so.