OpenStack Compute (nova)

AMQP issue , not sure if this a bug ...

Asked by Hugo Kou

Nova version : nova-api 2011.3~d3~20110630.1238-0ubuntu0ppa1~natty1
RabbitMQ : rabbitmq-server 2.3.1-1ubuntu1

root@nova:/var/log/nova# euca-run-instances -t m1.tiny -k hugo ami-003
UnknownError: An unknown error has occurred. Please try your request again.

========nova-api.log========
2011-07-01 14:50:32,166 DEBUG nova.rpc [-] Making asynchronous cast on scheduler... from (pid=2750) cast /usr/lib/pymodules/python2.7/nova/rpc.py:555
2011-07-01 14:50:32,167 ERROR nova.api [TTM-QDNQPV0O2B9Y2AIB hugo pro1] Unexpected error raised: (406, u"PRECONDITION_FAILED - cannot redeclare exchange 'nova' in vhost '/' with different type, durable, internal or autodelete value", (40, 10), 'Channel.exchange_declare')
(nova.api): TRACE: Traceback (most recent call last):
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/api/ec2/__init__.py", line 320, in __call__
(nova.api): TRACE: result = api_request.invoke(context)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/api/ec2/apirequest.py", line 78, in invoke
(nova.api): TRACE: result = method(context, **args)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/api/ec2/cloud.py", line 956, in run_instances
(nova.api): TRACE: block_device_mapping=kwargs.get('block_device_mapping', {}))
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/api.py", line 420, in create
(nova.api): TRACE: instance_id=instance_id)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/api.py", line 346, in _ask_scheduler_to_create_instance
(nova.api): TRACE: "injected_files": injected_files}})
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc.py", line 558, in cast
(nova.api): TRACE: publisher = TopicPublisher(connection=conn, topic=topic)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc.py", line 352, in __init__
(nova.api): TRACE: super(TopicPublisher, self).__init__(connection=connection)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/carrot/messaging.py", line 657, in __init__
(nova.api): TRACE: self.declare()
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/carrot/messaging.py", line 669, in declare
(nova.api): TRACE: auto_delete=self.auto_delete)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/carrot/backends/pyamqplib.py", line 256, in exchange_declare
(nova.api): TRACE: auto_delete=auto_delete)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/channel.py", line 843, in exchange_declare
(nova.api): TRACE: (40, 11), # Channel.exchange_declare_ok
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/abstract_channel.py", line 105, in wait
(nova.api): TRACE: return amqp_method(self, args)
(nova.api): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/channel.py", line 273, in _close
(nova.api): TRACE: (class_id, method_id))
(nova.api): TRACE: AMQPChannelException: (406, u"PRECONDITION_FAILED - cannot redeclare exchange 'nova' in vhost '/' with different type, durable, internal or autodelet e value", (40, 10), 'Channel.exchange_declare')
(nova.api): TRACE:
2011-07-01 14:50:32,168 ERROR nova.api [TTM-QDNQPV0O2B9Y2AIB hugo pro1] Environment: {"HTTP_AUTHORIZATION": "AWS 1874a406-b1ff-4a8f-9684-d46810a70159:pro1:n+pE0RgAgx6TP yrxm0K2uxnT3jQ=", "CONTENT_TYPE": "application/x-www-form-urlencoded; charset=UTF-8", "SCRIPT_NAME": "/services/Cloud", "REQUEST_METHOD": "POST", "HTTP_HOST": "172.16.4 .184:8773", "PATH_INFO": "/", "SERVER_PROTOCOL": "HTTP/1.0", "HTTP_USER_AGENT": "Boto/1.9b (linux2)", "SERVER_NAME": "172.16.4.184", "REMOTE_ADDR": "172.16.4.184", "HTT P_DATE": "Fri, 01 Jul 2011 06:50:31 GMT", "wsgi.url_scheme": "http", "SERVER_PORT": "8773", "GATEWAY_INTERFACE": "CGI/1.1", "HTTP_ACCEPT_ENCODING": "identity"}
2011-07-01 14:50:32,168 ERROR nova.api [TTM-QDNQPV0O2B9Y2AIB hugo pro1] UnknownError: An unknown error has occurred. Please try your request again.
2011-07-01 14:50:32,169 INFO nova.api [TTM-QDNQPV0O2B9Y2AIB hugo pro1] 0.366498s 172.16.4.184 POST /services/Cloud/ CloudController:RunInstances 400 [Boto/1.9b (linux2) ] application/x-www-form-urlencoded text/xml

=============================

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Steve Brown
Solved:
Last query:
Last reply:
Revision history for this message
Steve Brown (jpgeek) said : 		#1

I just got the same thing. Apparently rabbitmq is down and I am getting the same error in that log. I am running trunk 2011.3~d3~20110630.1238-0ubuntu0ppa1~natty1.

Revision history for this message
Best Steve Brown (jpgeek) said : 		#2

OK - it looks like this is a bug, or at least it is bombing out on the Jenkins tests also:

http://jenkins.openstack.org/job/nova-vpc/23/console

In the output the same error is occurring:

(nova.api.openstack): TRACE: AMQPChannelException: (406, u"PRECONDITION_FAILED - cannot redeclare exchange 'nova' in vhost '/nova' with different type, durable or autodelete value", (40, 10), 'Channel.exchange_declare')

Revision history for this message
Hugo Kou (tonytkdk) said : 		#3

Thanks Steve Brown, that solved my question.

Revision history for this message
Ram Durairaj (dramesh) said : 		#4

I'm facing this error after my install on Ubuntu 11.04 with cloud builders script based installation pointing to latest nova branch.

Is this bug fixed?

Revision history for this message
Edier Z (edalzap) said : 		#5

I got this same error in Ubuntu 11.10 Server 64 bits, Any hint how to fix it?

Thanks.
Error Output in nova-compute.log:

2011-10-18 12:53:45,072 CRITICAL nova [-] (406, u"PRECONDITION_FAILED - cannot redeclare exchange 'compute_fanout' in vhost '/' with different type, durable, internal or autodelete value", (40, 10), 'Channel.exchange_declare')
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-compute", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 161, in start
(nova): TRACE: self.conn.create_consumer(self.topic, self, fanout=True)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 494, in create_consumer
(nova): TRACE: self.declare_fanout_consumer(topic, ProxyCallback(proxy))
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 457, in declare_fanout_consumer
(nova): TRACE: self.declare_consumer(FanoutConsumer, topic, callback)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 394, in declare_consumer
(nova): TRACE: self.consumer_num.next())
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 210, in __init__
(nova): TRACE: **options)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 61, in __init__
(nova): TRACE: self.reconnect(channel)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 68, in reconnect
(nova): TRACE: self.queue.declare()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 359, in declare
(nova): TRACE: return (self.name and self.exchange.declare(nowait),
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 151, in declare
(nova): TRACE: nowait=nowait)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 14, in blocking
(nova): TRACE: return __sync_current(fun, *args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 40, in __eblocking__
(nova): TRACE: return spawn(fun, *args, **kwargs).wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 843, in exchange_declare
(nova): TRACE: (40, 11), # Channel.exchange_declare_ok
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
(nova): TRACE: return self.dispatch_method(method_sig, args, content)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
(nova): TRACE: return amqp_method(self, args)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
(nova): TRACE: (class_id, method_id))
(nova): TRACE: AMQPChannelException: (406, u"PRECONDITION_FAILED - cannot redeclare exchange 'compute_fanout' in vhost '/' with value", (40, 10), 'Channel.exchange_declare')
(nova): TRACE:

Revision history for this message
Gregor Beslic (gregor-beslic) said : 		#6

The Exchange is probably declared with a different set of parameters (auto-deletion, etc.) in the new version of OpenStack and RabbitMq is quite "nazi" about this sort of stuff. There's a nice way and a not-so-nice way to manually delete the Exchange.

THE NICE WAY:
If the Queues binded to the Exchange are empty, the Exchange can be deleted using the following commands:
> rabbitmqctl stop_app
> rabbitmqctl reset
> rabbitmqctl start_app

2) List the Exchanges to see if the it was removed from the list:
> rabbitmqctl list_exchanges

If it remains on the list, use "amqp-utils" Ruby Gem to do the job, since "rabbitmqctl" does not support Exchange deletion.
THE NOT-SO-NICE WAY:
> apt-get install ruby rubygems1.8
> gem install amqp-utils
> cd /var/lib/gems/1.8/gems/amqp-utils-0.5.1/bin # the path might be different on your system
> ./amqp-delexch <EXCHANGE_NAME>

Double-check by retrieving the Exchange list once again:
> rabbitmqctl list_exchanges

It should now be gone...