sys:Resources calls cause murano to never finish

Asked by smarta94

No matter what type I deploy to an environment -- aside from a simple VM with absolutly no additional items beyond the default OS image -- murano's environment seems to stall out. The instances get created and are accessible, but they are basic images with no addtions that the type is supposed to have. In the murano environment, it seems stuck at Deploy in Progress. Heat stack shows it as built, and because it is still in progress it its not possible to cancel or delete the environment, and this causes issues with usability eventually when many environments are in this state. It seems to get stuck any time the
$resources: new(sys:Resources); - $template: $resources.yaml('<pickyourtemplate>.template')...
gets called. Whether its a simple Tomcat install or etcd in Kubernetes Cluster, the same happens and nothing appears in the logs to indicate what the problem is.

How do I go about deleting the environments / what could be causing this to happen?

I am using Juno 2014.2.1-1 for openstack, and muranoclient (python) is 0.5.9.

Question information

Language:
English Edit question
Status:
Solved
For:
Murano Edit question
Assignee:
Ekaterina Chernova Edit question
Solved by:
Stan Lagun
Solved:
Last query:
Last reply:
Revision history for this message
Ekaterina Chernova (efedorova) said :
#1

Hi!

That's a known behavior for Juno version. Currently we adding the opportunity to delete hanged environment, but now it's can be done only through deleting envs and sessions directly at the database.

We also have timeouts feature in kilo, so the deployment will automatically fail after some time.

In your case it looks like there are no connection between murano-agent and murano-engine. Did you set up correct rabbitmq setting in murano config at [rabbitmq] section? Did you install murano-agent to an image, with witch you are deploying?

Is there a murano-agent log file at /var/log/ directory on the spawned vm?

Revision history for this message
Serg Melikyan (smelikyan) said :
#2

Can you provide engine log for deployment during which this happens?

Revision history for this message
Best Stan Lagun (slagun) said :
#3

The most probable reason for such behavior is that something wrong with murano agent. It may be that you are using image without murano agent or there is something wrong with metadata service or maybe you haven't configured [rabbitmq] section in your config file and murano agent cannot connect to RabbitMQ. Till kilo we hadn't agent timeouts. Because Murano <-> agent communications are indirect (via RabbitMQ) Murano never knows if agent on the other side exists at all. So it just waits for it to send response and this can take forever if agent wasn't properly configured

Revision history for this message
smarta94 (smarta94) said :
#4

I must have overlooked the murano agent part of the install guide. Anyway, I attempted to get murano-agent installed using diskimage-builder as in the readme file on github. I get this as a result shortly after it runs:

mount: unknown filesystem type 'LVM2_member'
device-mapper: remove ioctl on loop5p2 failed: Device or resource busy
loop deleted : /dev/loop5

but when I do a vgscan it finds it just fine (vg_centos) uising metadata type lvm2.
lvs shows its volumes just fine.

I do have another mounted lv that hosts my OS on the system, but I wouldn't think that would be causing any problems. I cannot seem to find any notes in diskbuilder about this. I can manually mount the volume it disk-image-create makes with no problems, and look through the files. Is there a way to manually install murano-agent aside from diskbuilder?

Revision history for this message
smarta94 (smarta94) said :
#5

And to configure the murano agent conf file, as this is on a vm I assume I still pass in the acutal username/password for rabbit rather than guest/guest as it defaults to. And what would go in for the host? My rabbit hosts or being a vm within murno does this change?

Revision history for this message
Serg Melikyan (smelikyan) said :
#6

smarta94, you don't need to provide configuration file in image since it's done during spawning VM via cloud-init by Murano.

Revision history for this message
Stan Lagun (slagun) said :
#7

smarta94, you don't need to configure agent on VM. Instead you need to provide correct connection parameters in [rabbitmq] section in murano.conf on the server. And it must not be localhost or 127.0.0.1. You also cannot use guest/guest as guest account can be used only from localhost

Revision history for this message
smarta94 (smarta94) said :
#8

When it spawns it gives it an ip of the management network, which is inaccessible by the vm -- I need it to be given the public api address to communicate with rabbit. However, my rabbit is configured for everything else on the management network (192 is public, 10... is the management that everything else points to for rabbit). If I have this set, it doesn't appear to find a queue (404: (NOT_FOUND - no queue 'agent-tasks' in vhost '/', (60, 20), None)).
Is it possible to use the 192 network on the vms while pointing other stuff to the management 10.1... network? Or does everything have to point to the same address on rabbit (despite being able to have rabbit listen on both)?

Revision history for this message
Stan Lagun (slagun) said :
#9

We usually have separate RabbitMQ installation for engine <--> VM communications that is not in the management network. Thats why there are 2 sets of options for RabbitMQ in murano.conf - one for API <--> engine RabbitMQ which is typically the one that all OpenStack services and [rabbitmq] section for a dedicated Murano RabbitMQ

Revision history for this message
smarta94 (smarta94) said :
#10

Yeah, I just found that after I posted my last response. I am working on configuring my images again to test that out. Thanks.

Revision history for this message
smarta94 (smarta94) said :
#11

So it appears that this is solved -- but one last question, is it recommended to use a separate rabbit userid/pass for the murano-agent interaction? Thanks Again!

Revision history for this message
smarta94 (smarta94) said :
#12

Thanks Stan Lagun, that solved my question.

Revision history for this message
Stan Lagun (slagun) said :
#13

If you have separate RabbitMQ instance dedicated to agent communications it doesn't matter what password you use. If it is shared instance then yes, it is better to have separate credentials