neutron

3 nodes are involved: controller, network gateway, compute node. New instances are not communicating at all, no ip addresses.

Asked by Aryan

Seems something terribly wrong, but before starting from scratch it would be nice if someone give me some hints :)

Previously everything was working with nova-network (FlatDHCP) on the controller node.

Now 3 nodes are involved: controller, network gateway, compute node. New instances are not communicating at all, no ip addresses.

- Each node has two interfaces, eth0: management, eth1: data

- Namespace is not disabled

Controller:
services: quantum-server, nova-* (but not nova-network), openvswitch-switch
ifconfig: http://pastebin.com/8gvGAQLd
nova.conf: http://pastebin.com/TzNpMR56
ovs-vsctl show: http://pastebin.com/xrD5u1hX
quantum.conf: http://pastebin.com/S9YSKCZ5
ovs_quantum_plugin.ini: http://pastebin.com/KciZ4bin
[LOGS]
quantum-server.log: http://paste.openstack.org/show/24617/

Network Gateway:
services: quantum-l3-agent, quantum-dhcp-agent, quantum-plugin-openvswitch-agent, openvswitch-switch
dhcp_agent.ini: http://pastebin.com/2pWVSqqE
l3_agent.ini: http://pastebin.com/jNTbQUwG
ovs_quantum_plugin.ini: http://pastebin.com/r1gPrXLF
ovs-vsctl show: http://pastebin.com/Wuh9r8B7 (and here we go for the strange output without any gw-xxx interface. Note: I created two networks)

This one has only two NICs, so I haven't assigned any interface for external network. It should still work?
[LOGS]
dhcp-agent.log: http://paste.openstack.org/show/24609/
openvswitch-agent.log: http://paste.openstack.org/show/24613/

Compute Node:
services: nova-compute, nova-api-metadata, quantum-plugin-openvswitch-agent
ovs-vsctl show: http://pastebin.com/qd6kfNpJ
ovs_quantum_plugin.ini: http://pastebin.com/rZVwSxf2
[LOGS]
openvswitch-agent.log: http://paste.openstack.org/show/24616/

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
Rabi Mishra
Solved:
Last query:
Last reply:
Revision history for this message
dan wendlandt (danwent) said : 		#1

what topology are you trying to achieve? Does it map to one of the use cases in the admin guide, such as http://docs.openstack.org/trunk/openstack-network/admin/content/use_cases_single_router.html

can you provide the output from the following commands:

quantum net-list
quantum router-list
quantum port-list -c id -c fixed_ips -c device_owner

thanks.

Revision history for this message
Aryan (aryantaheri) said : 		#2

Dan,

Thank you for the feedback.

Yes, I used "Per-tenant Routers with Private Networks".

Unfortunately, I started it from scratch yesterday. But still I have the DB dump:
http://paste.openstack.org/show/24956/

I'll come back to you with more information, when the new configuration is in place.

Cheers.

Revision history for this message
Aryan (aryantaheri) said : 		#3

The new configuration is almost the same as previous one. No success yet.

quantum net-list
quantum router-list
quantum port-list -c id -c fixed_ips -c device_owner
output:
http://paste.openstack.org/show/25016/

[Network Gateway]
# ovs-vsctl show
http://paste.openstack.org/show/25017/

# ip netns list
http://paste.openstack.org/show/25018/

# ip netns exec qdhcp-e66b8a22-daff-4850-9656-282bee68819d ifconfig
http://paste.openstack.org/show/25019/

[Compute Node]
# ovs-vsctl show
http://paste.openstack.org/show/25021/

If I assign the VM's IP manually (10.10.11.4), VM can retrieve the mac address for 10.10.11.2. Now when I ping there is no reply, BUT tcpdump output on both sides show something different !!!

From network gateway to the vm: http://paste.openstack.org/show/25035/
From vm to the network gateway: http://paste.openstack.org/show/25036/

One more thing which may help:
I specified vlans from 200 to 250 in the config file. The vlan_allocation table confirms that. But the tag field in "ovs-vsctl show" output shows tag 1. This should not be true. (Using HP ProCurve 2910al-48g, corresponding ports are also on the DEFAULT_VLAN 1, in addition to 200-250)

Thanks.

Revision history for this message
Rabi Mishra (mishra-rabi) said : 		#4

Hi Aryan,

Did u manage to find a solution to your issue? I am stuck with the same issue.

thanks,
Rabi

Revision history for this message
Aryan (aryantaheri) said : 		#5

Rabi,

Not yet, I'll continue working on it next week. I'll update the question in case of any success.

Revision history for this message
Best Rabi Mishra (mishra-rabi) said : 		#6

I did manage to fix it.. I have used vlans 2001:4094 in my ovs plugin
'physnet:2001:4094'. However, I have not created those VLANs on the switch.
My ports in the switch are tunked for all VLANs '1:4094'. Once I created a
few VLANs used by my logical networks (2001 and 2003), it worked like a
charm.ch

The VLAN tages you see with 'ovs-vsctl show' on the compute nodes show 1,2
etc, but they get mapped to 2001 when go beyond the soft switch.

The other thing I have done is changed my vif drivers from what has been
mentioned in many installation
guides (libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver)
 to

libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver
.

Hope this helps you.

Thanks,
Rabi

On Thu, Nov 22, 2012 at 1:45 PM, Aryan <<email address hidden>
> wrote:

> Question #213459 on quantum changed:
> https://answers.launchpad.net/quantum/+question/213459
>
> Aryan posted a new comment:
> Rabi,
>
> Not yet, I'll continue working on it next week. I'll update the question
> in case of any success.
>
> --
> You received this question notification because you are a direct
> subscriber of the question.
>

--
Regards,
Rabi Mishra

Revision history for this message
Aryan (aryantaheri) said : 		#7

Well done, I'll check it out.

Just a terminology question, by "trunk" you mean Cisco trunking, not HP trunking? So in HP term, you've tagged those ports, or just left them untouched?

Thanks
Aryan

Revision history for this message
Rabi Mishra (mishra-rabi) said : 		#8

Yeah I have connected the private network to a cisco 3650 switches. Not
sure of commands for HP... I think ports are vlan tagged by default...

Switch(config)# *interface xx*

Switch(config-if)# *switchport trunk encapsulation dot1q*
Switch(config-if)# *switchport mode trunk*

On Thu, Nov 22, 2012 at 3:01 PM, Aryan <<email address hidden>
> wrote:

> Question #213459 on quantum changed:
> https://answers.launchpad.net/quantum/+question/213459
>
> Aryan posted a new comment:
> Well done, I'll check it out.
>
> Just a terminology question, by "trunk" you mean Cisco trunking, not HP
> trunking? So in HP term, you've tagged those ports, or just left them
> untouched?
>
> Thanks
> Aryan
>
> --
> You received this question notification because you are a direct
> subscriber of the question.
>

--
Regards,
Rabi Mishra

Revision history for this message
Aryan (aryantaheri) said : 		#9

Rabi, Thank you

It was extremely strange, changing the "libvirt_vif_driver" to "nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver" solved the problem.

--Aryan

Revision history for this message
Aryan (aryantaheri) said : 		#10

Thanks Rabi Mishra, that solved my question.

Revision history for this message
dan wendlandt (danwent) said : 		#11

changing the libvirt_vif_driver in that way basically makes it so that nova security groups are not applied. so it may have been that you could not communicate due to the fact that nova security groups were blocking the traffic.

Revision history for this message
Aryan (aryantaheri) said : 		#12

Dan,

It was not about ICMP, or SSH. The VM didn't get an IP address, DHCP requests were sent, I don't remember how far they got, but the network gateway didn't receive them.

I may change it back and check out how far they get.