Network configuration / Can't ping or ssh to our instances

If you would like to catch me on AIM or Skype we can work together to get this figured out or find someone who can. My name on both services is "JordanRinke".

-----Original Message-----
From: "Everett Toews" <email address hidden>
Sent: Thursday, February 17, 2011 6:58pm
To: <email address hidden>
Subject: [Question #145820]: Network configuration / Can't ping or ssh to our instances

New question #145820 on OpenStack Compute (nova):
https://answers.launchpad.net/nova/+question/145820

We have been trying to deploy nova (Bexar) for the past week on a single machine. We can install and configure nova on a fresh Ubuntu 10.10 server and even successfully start VMs but we have not been able to connect to those VMs. Neither ping nor ssh work (yes, we have done the euca-authorize steps).

We've tried all of the steps from the http://wiki.openstack.org/TroubleshootingNova. We've tried asking other questions, https://answers.launchpad.net/nova/+question/145062 and https://answers.launchpad.net/nova/+question/145063 (no replies). We've tried going on IRC, giving bits and pieces of information about our problems and getting good troubleshooting tips in return but, ultimately, nothing that has worked for us. We've read about a similar problems: https://answers.launchpad.net/nova/+question/141960 was already solved and the patch merged in Bexar and https://answers.launchpad.net/nova/+question/145567 doesn't seem to be relevant anymore (can't find --flat_interface reference in any of the nova-<service> --help).

So, out of desperation, here is all of the information about our configuration (IP addresses changed) in the hopes that someone can pinpoint what the problem is.

Is our networking configured properly for Nova? Why can't we ping or ssh to our instances?

Any answers or network troubleshooting tips are greatly appreciated!

********** Installation Architecture **********
By design our network architecture very closely resembles the Nova example installation architecture.

http://docs.openstack.org/openstack-compute/admin/content/figures/NOVA_install_arch.png

Keep in mind that, at the moment, we are only trying to install to a single machine.

********** Installation Procedure **********
We are doing the scripted installation (http://docs.openstack.org/openstack-compute/admin/content/ch03s02.html#d5e161). When we run nova-CC-install-v1.1.sh here are the values we input:

S3 Host IP (Default is 99.99.99.185 -- ENTER to accept):
RabbitMQ Host IP (Default is 99.99.99.185 -- ENTER to accept):
Cloud Controller Host IP (Default is 99.99.99.185 -- ENTER to accept):
mySQL Host IP (Default is 99.99.99.185 -- ENTER to accept):

Controller network range for ALL projects (normally x.x.x.x/12):192.168.0.0/16
Total amount of usable IPs for ALL projects:64
Nova project user name:admin
Nova project name:admin-project
Desired network + CIDR for project (normally x.x.x.x/24):192.168.0.0/24
How many networks for project:1
How many availible IPs per project network:8

Please enter your local server IP (Default is 99.99.99.185 -- ENTER to accept):
Please enter your broadcast IP (Default is 99.99.99.191 -- ENTER to accept):
Please enter your netmask (Default is 255.255.255.224 -- ENTER to accept):
Please enter your gateway (Default is 99.99.99.161 -- ENTER to accept):
Please enter your default nameserver (Default is 99.99.99.194 -- ENTER to accept):

********** Installation Output **********
Installing packages
###################
Installing package 'python-software-properties' ... ok
Installing package 'python-mysqldb' ... ok
Installing package 'mysql-server' ... ok
Installing package 'nova-api' ... ok
Installing package 'nova-network' ... ok
Installing package 'nova-objectstore' ... ok
Installing package 'nova-scheduler' ... ok
Installing package 'nova-compute' ... ok
Installing package 'unzip' ... ok
Installing package 'vim' ... ok
Installing package 'euca2ools' ... ok
Finalizing mySQL setup
######################

...done...

Setting up Nova configuration files
###################################
Initializing database
...done...

...done...

Generating Nova credentials
###########################
...done...

Creating br100 bridge device
############################

...done...

Restarting networking
 * Reconfiguring network interfaces... ssh stop/waiting
ssh start/running, process 4472
ssh stop/waiting
ssh start/running, process 4500
ssh stop/waiting
ssh start/running, process 4562
                                                                                                                                                                                                                                     [ OK ]
...done...

#################NOTE#####NOTE#####NOTE#####NOTE#####NOTE#####NOTE#################
#Be sure to source your credential file into your environment after config changes#
###########################e.g. source /root/creds/novarc##########################

Ensure all five Nova services are running
#########################################
root 3763 0.0 0.0 12228 1884 pts/0 S+ 17:15 0:00 /bin/bash ./nova-CC-install-v1.1.sh
nova 4623 0.0 0.0 35632 1264 ? Ss 17:17 0:00 su -c nova-network --flagfile=/etc/nova/nova.conf nova
nova 4625 6.8 0.3 105064 27180 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-network --flagfile=/etc/nova/nova.conf
nova 4637 0.0 0.0 35632 1268 ? Ss 17:17 0:00 su -c nova-compute --flagfile=/etc/nova/nova.conf nova
nova 4639 13.2 0.4 157360 38864 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-compute --flagfile=/etc/nova/nova.conf
nova 4662 0.0 0.0 35632 1264 ? Ss 17:17 0:00 su -c nova-api --flagfile=/etc/nova/nova.conf nova
nova 4663 16.5 0.2 80224 23948 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf
nova 4676 15.5 0.3 96716 24604 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-objectstore --uid 106 --gid 65534 --pidfile /var/run/nova/nova-objectstore.pid --flagfile=/etc/nova/nova.conf --nodaemon --logfile=/var/log/nova/nova-objectstore.log
nova 4692 0.0 0.0 35632 1268 ? Ss 17:17 0:00 su -c nova-scheduler --flagfile=/etc/nova/nova.conf nova
root 4694 0.0 0.0 8952 888 pts/0 S+ 17:17 0:00 grep -i nova
nova 4695 0.0 0.0 27948 4408 ? R 17:17 0:00 /usr/bin/python /usr/bin/nova-scheduler --flagfile=/etc/nova/nova.conf

Setup default ICMP and SSH access to your future VMs
####################################################
Allowing ping and SSH to your running instances
...done...

######################################################################
#You /MUST/ re-source your 'novarc' to use the API commands since the#
##script cannot pass the source information out of it's own process###
######################################################################

dnsmasq: no process found

The next thing you are going to want to do it get a VM to test with. You can find a test VM how-to, and read more about custom image creation at "http://nova.openstack.org/adminguide/multi.node.install.html" and "http://wiki.openstack.org/GettingImages

Enjoy your new private cloud!

********** Running Instances Procedure **********
We followed the instructions from Starting Images (http://docs.openstack.org/openstack-compute/admin/content/ch05s01.html). The instance runs and here is the result of euca-describe-instances,

RESERVATION r-et60ec7l admin-project default
INSTANCE i-00000001 ami-3if67tvi 192.168.0.2 192.168.0.2 running admin (admin-project, ubuntu) 0 m1.tiny 2011-02-17 22:34:38 nova

********** Ping and SSH **********
root@ubuntu:~# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
^C
--- 192.168.0.2 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3007ms

root@ubuntu:~# ssh 192.168.0.2
ssh: connect to host 192.168.0.2 port 22: Connection timed out

********** traceroute **********
root@ubuntu:~# traceroute 192.168.0.2
traceroute to 192.168.0.2 (192.168.0.2), 30 hops max, 60 byte packets
send: Operation not permitted

********** euca-get-console-output **********
As described in these other questions, https://answers.launchpad.net/nova/+question/145062 and https://answers.launchpad.net/nova/+question/145063

********** Logs **********
It is worthwhile to note there are no ERRORs in the logs.

********** nova.conf **********
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--verbose
--s3_host=99.99.99.185
--rabbit_host=99.99.99.185
--cc_host=99.99.99.185
--ec2_url=http://99.99.99.185:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=64
--FAKE_subdomain=ec2
--routing_source_ip=99.99.99.185
--verbose
--sql_connection=mysql://root:nova@99.99.99.185/nova
--network_manager=nova.network.manager.FlatManager

********** ifconfig **********
br100 Link encap:Ethernet HWaddr 00:1b:78:d2:ab:9a
          inet addr:99.99.99.185 Bcast:99.99.99.191 Mask:255.255.255.224
          inet6 addr: fe80::21b:78ff:fed2:ab9a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:145536 errors:0 dropped:0 overruns:0 frame:0
          TX packets:94481 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:380460510 (380.4 MB) TX bytes:6805858 (6.8 MB)

eth0 Link encap:Ethernet HWaddr 00:1b:78:d2:ab:9a
          inet6 addr: fe80::21b:78ff:fed2:ab9a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:271190 errors:0 dropped:0 overruns:0 frame:0
          TX packets:95283 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:391928462 (391.9 MB) TX bytes:7236250 (7.2 MB)
          Interrupt:16

eth1 Link encap:Ethernet HWaddr 00:1b:78:d2:ab:9b
          inet addr:192.168.3.185 Bcast:192.168.255.255 Mask:255.255.0.0
          inet6 addr: fe80::21b:78ff:fed2:ab9b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:49 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3136 (3.1 KB) TX bytes:1418 (1.4 KB)
          Interrupt:17

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:1020865 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1020865 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1738335671 (1.7 GB) TX bytes:1738335671 (1.7 GB)

virbr0 Link encap:Ethernet HWaddr 06:99:07:25:0b:9c
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          inet6 addr: fe80::499:7ff:fe25:b9c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:468 (468.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:2b:32:49
          inet6 addr: fe80::fc16:3eff:fe2b:3249/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:720 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1635 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:30456 (30.4 KB) TX bytes:98504 (98.5 KB)

********** iptables-save **********
# Generated by iptables-save v1.4.4 on Thu Feb 17 18:27:59 2011
*nat
:PREROUTING ACCEPT [276:35596]
:OUTPUT ACCEPT [380:24818]
:POSTROUTING ACCEPT [386:25062]
:SNATTING - [0:0]
-A PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 99.99.99.185:8773
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -j ACCEPT
-A POSTROUTING -s 10.0.0.0/8 -d 10.128.0.0/24 -j ACCEPT
-A POSTROUTING -j SNATTING
-A SNATTING -s 10.0.0.0/8 -j SNAT --to-source 99.99.99.185
COMMIT
# Completed on Thu Feb 17 18:27:59 2011
# Generated by iptables-save v1.4.4 on Thu Feb 17 18:27:59 2011
*filter
:INPUT ACCEPT [643452:1525282513]
:FORWARD ACCEPT [8:352]
:OUTPUT ACCEPT [643396:1525477930]
:nova-fallback - [0:0]
:nova-inst-1 - [0:0]
:nova-local - [0:0]
:nova-sg-1 - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -j nova-local
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j nova-local
-A nova-fallback -j DROP
-A nova-inst-1 -m state --state INVALID -j DROP
-A nova-inst-1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-inst-1 -j nova-sg-1
-A nova-inst-1 -s 192.168.0.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A nova-inst-1 -s 192.168.0.0/29 -j ACCEPT
-A nova-inst-1 -j nova-fallback
-A nova-local -d 192.168.0.2/32 -j nova-inst-1
-A nova-sg-1 -p icmp -j ACCEPT
-A nova-sg-1 -p tcp -m tcp --dport 22 -j ACCEPT
COMMIT
# Completed on Thu Feb 17 18:27:59 2011

********** brctl show **********
bridge name bridge id STP enabled interfaces
br100 8000.001b78d2ab9a no eth0
       vnet0
virbr0 8000.000000000000 yes

********** TL;DR **********
Is our networking configured properly for Nova? Why can't we ping or ssh to our instances?

You received this question notification because you are an answer
contact for OpenStack Compute (nova).

connect to the running instance through the serial console and look if the network in the instance is coming up.

Definitely willing to give it a try but I've never accessed a VM via the serial console before. How would I go about doing that?

We're using KVM by the way.

Thanks,
Everett

On Feb 18, 2011, at 5:13 AM, Christian Berendt <email address hidden> wrote:

> Your question #145820 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/145820
>
> Christian Berendt posted a new comment:
> connect to the running instance through the serial console and look if
> the network in the instance is coming up.
>
> --
> You received this question notification because you are a direct
> subscriber of the question.

On a suggestion from Jordan, I switched to FlatDHCPManager for the networking mode and that fixed this problem.

To /etc/nova/nova.conf I added:
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_dhcp_start=192.168.0.2
--public_interface=eth0
--flat_interface=eth1
--flat_injected=False

I also removed the bridge from /etc/network/interfaces and restarted networking:
/etc/init.d/networking restart

Then I restarted all of the nova services:
restart libvirt-bin
restart nova-network
restart nova-compute
restart nova-api
restart nova-objectstore
restart nova-scheduler

Hi Everett Toews
  I am seeing the same problem as your. I am glad to hear that your is resolved

Can you help me?

Can you tell me which installation process you had followed?
1: NovaInstall/DevInstallScript??
2:NovaInstall/DevPkgInstall??
3:NovaInstall/Bexar??

I tried Bexar and DevInstallScript -- not sure when went wrong - I could not connect.

When tried to start nova-xyz services -- they dont throw up errors. However - no logs are produced + every time I try to do "ps -aef | grep nova" -- I am seeing different process IDs -- this means something wrong.

Could you please help me?

I am redoing the entire steps again as mentioned in - -I am seeing some issues
0:
http://docs.openstack.org/openstack-compute/admin/content/ch03s02.html

1:

Installing packages
###################
Installing package 'python-software-properties' ... Already installed
Installing package 'python-mysqldb' ... Already installed
Installing package 'mysql-server' ... Already installed
Installing package 'nova-api' ... ok
Installing package 'nova-network' ... ok
Installing package 'nova-objectstore' ... Already installed
Installing package 'nova-scheduler' ... ok
Installing package 'nova-compute' ... ok
Installing package 'unzip' ... Already installed
Installing package 'vim' ... Already installed
Installing package 'euca2ools' ... Already installed
Finalizing mySQL setup
######################

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
...done...

Notice the errors

3:the script took long time at
enabling SSH to running instances...lot of time - -over 30 mins etc

4: Notice I am trying to connect to Amazon EC2 as my Cloud Controller + S3

Please help me find the root of the issue reported in #2 and why delay in #3.

Will this Script work - if my Cloud Controller and S3 EC2_URL is AMAZON EC2? I have all the required info like
EC2 CERT
EC2 Secrete Key
Access Key
Access Secret key

Hi Sunny,

Sorry I missed your question until now. If you haven't solved your problem already, I would advise starting a new question as this one has be solved already (in my case). It's best to have separate questions for these types of problems. Even though the symptoms of the problem are similar the answer may be very different.

Thanks,
Everett

I meet the same issue in the single node installation with one NIC enabled. how do I fix this issue? thanks a lot, Further, I have only 10.x.x.x subnet, not 192.168.x.x subnet.
here is my nova.conf:
======

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--s3_host=10.101.1.142
--rabbit_host=10.101.1.142
--cc_host=10.101.1.142
--ec2_url=http://10.101.1.142:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=64
--FAKE_subdomain=ec2
--routing_source_ip=10.101.1.142
--verbose
--sql_connection=mysql://root:iforgot@10.101.1.142/nova
--network_manager=nova.network.manager.FlatManager

moreover, we have one dhcp and gateway :10.0.0.1 and are not allow to have another dhcp server since they will conflict in the same network.

how do I config openstack's network to avoid this issue in my case ?

the error message of console log is below:
======
2011-07-11 10:04:11,496 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id

2011-07-11 10:04:11,498 - DataSourceEc2.py[WARNING]: 10:04:11 [ 1/100]: url error [[Errno 113] No route to host]