Instance metadata not found (http error [404]) for instances started on a compute node

When I remove the flag --ec2_dmz_host from the compute node the console output becomes:

cloud-init start-local running: Mon, 27 Feb 2012 20:54:39 +0000. up 1.91 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 192.168.22.41 255.255.255.0 02:16:3e:52:1f:dd
ci-info: route-0: 0.0.0.0 192.168.22.20 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.22.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Mon, 27 Feb 2012 20:54:53 +0000. up 15.14 seconds
2012-02-27 20:55:03,278 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]
2012-02-27 20:55:04,282 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]

So it seems in the original setup, with the --ec2_dmz_host flag, instances can reach the api server. Why can't find metadata?

Your traffic is probably being snatted, so the metadata server gets the wrong source ip and can't look up info. Try setting
--dmz_cidr to the ip address of your api host
e.g. --dmz_cidr=192.168.22.1/32 (or wherever your api host is running)

On Feb 27, 2012, at 1:00 PM, Ahmad Al-Shishtawy wrote:

> Question #189026 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/189026
>
> Ahmad Al-Shishtawy gave more information on the question:
> When I remove the flag --ec2_dmz_host from the compute node the console
> output becomes:
>
> cloud-init start-local running: Mon, 27 Feb 2012 20:54:39 +0000. up 1.91 seconds
> no instance data found in start-local
> ci-info: lo : 1 127.0.0.1 255.0.0.0
> ci-info: eth0 : 1 192.168.22.41 255.255.255.0 02:16:3e:52:1f:dd
> ci-info: route-0: 0.0.0.0 192.168.22.20 0.0.0.0 eth0 UG
> ci-info: route-1: 192.168.22.0 0.0.0.0 255.255.255.0 eth0 U
> cloud-init start running: Mon, 27 Feb 2012 20:54:53 +0000. up 15.14 seconds
> 2012-02-27 20:55:03,278 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]
> 2012-02-27 20:55:04,282 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]
>
> So it seems in the original setup, with the --ec2_dmz_host flag,
> instances can reach the api server. Why can't find metadata?
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

When I set --dmz_cidr=<api host ip>/32 I get:

2012-02-27 21:35:50,279 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [timed out]

When I set --dmz_cidr=192.168.22.1/32 I get same error:
2012-02-27 21:41:28,384 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]

I forgot to mention that I have one nic (eth0)

Same issue here, after I added iptables nova-network-PREROUTING

cloud-init start-local running: Thu, 01 Mar 2012 07:59:09 +0000. up 1.70 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 10.100.80.3 255.255.255.0 02:16:3e:6f:92:f3
ci-info: route-0: 0.0.0.0 10.100.80.1 0.0.0.0 eth0 UG
ci-info: route-1: 10.100.80.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Thu, 01 Mar 2012 07:59:11 +0000. up 3.03 seconds
2012-03-01 07:59:11,365 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-03-01 07:59:12,433 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-03-01 07:59:13,509 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-03-01 07:59:14,577 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]

For multi-node installation, the root cause for both http error[404] or url error are, I think, request bad url upon vm initialization when starting up on compute mode, which is resulted from bad nova.conf configuration, esp. the network config.

When I changed nova.conf at each of the compute node as follows, all vms can be set up correctly on each of node, and ssh to the vm would be fine as well.

--network_manager=nova.network.manager.FlatDHCPManager
--fixed_range=10.100.80.0/24
--network_size=256
--flat_network_dhcp_start=10.100.80.2
--public_interface=eth1
--flat_interface=eth0
--flat_network_bridge=br100
--flat_injected=false

P.S., the scenario for my deployment is as fallows,
control: 10.100.79.136
compute node: 10.100.79.136, 10.100.79.137.

and I created the private network as:
nova-manage network create private --fixed_range_v4=10.100.80.0/24 --num_networks=1 --network_size=256 --multi_host=T

See a possible solution here:

http://askubuntu.com/questions/165115/ubuntu-cloud-instances-in-openstack-cant-import-public-ssh-key

Basically, you should try restarting nova-api and rebooting your instance:

    $ sudo restart nova-api