OpenStack Heat

How to fill context for rpc call when calling heat engine

Asked by Samuli Silvius

I have created heat extension which listens nova notifications "notifications.info" and does actions for instances in stacks accordingly (e.g. instance stopped could restart instance in stack). After received the notification I'm calling heat engine through RPC mechanism (i.e. using heat/heat/rpc/client.py and EngineClient class from there).

The problem is how to fill the context parameter correctly when calling heat engine? E.g. when I call
set_watch_state(ctxt, watch_name, state) i.e. want to set alarm (whihc has restarter policy in my template). How to fill ctxt parameter correctly concerning passwords?

I can get stack related context from heat db with

            rs = db_api.resource_get_by_physical_resource_id(None, instance_id)
            if not rs:
                LOG.debug("No stack found for stopped instance '%s'. Do nothing." % instance_id)
                return

            user_creds = db_api.user_creds_get(rs.stack.user_creds_id)
            stack_context = context.RequestContext.from_dict(user_creds)

with instanceId and then use stack_context successfully in any further db reads etc. But when using the same context in rpc call it fails to utf8 encoding as "password" and "service_password" are in some odd format when read from db.

Everything works if I in the code assign
stack_context.password = "root" #my admin password happens to be root
stack_context.service_password = "" # just set it empty to avoid utf8 encoding error

So how I get the password correctly?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Heat Edit question
Assignee:
No assignee Edit question
Solved by:
Samuli Silvius
Solved:
Last query:
Last reply:
Revision history for this message
Steven Hardy (shardy) said : 		#1

Basically, you need to either do this inside heat-engine (where we store encrypted credentials in the database for exactly this kind of scenario), or store credentials in your code.

Is there a reason why you aren't implementing this either in the main heat-engine code, or via a custom resource plugin?

See:

https://wiki.openstack.org/wiki/Heat/Plugins
and
https://github.com/openstack/heat/blob/master/heat/engine/service.py#L526

We would love to have improved support for nova notifications in heat, so it would be great if you could work with us and contribute to heat, instead of doing something standalone and accessing via the RPC API.

Note the RPC API is intended to be private, and may change at any time (we aren't planning to declare the RPC version final until the havana release), so if you go ahead with this approach, your code is likely to get broken, it would be much better to add this to heat-engine or use a custom resource plugin IMO, we do not support usage of the heat RPC interfaces in this way.

Revision history for this message
Jari Ojala (jari-j-ojala) said : 		#2

Is it true, that resource plugins are kind of "passive", i.e. they do not "run" constantly (for example in a dedicated thread, or part of event-loop), but are instead passively waiting to be invoked in order to do their time-constrained task?

If that's the case, it sounds that resource plugins wouldn't be as such suitable for continuous monitoring of nova-/ceilometer-/whatever notifications?

Would it be feasible to provide some sort of "listener plug-in" -support for heat-engine? Any preliminary ideas how to build one?

Revision history for this message
Angus Salkeld (asalkeld) said : 		#3

We want to remove _all_ constantly running jobs! I think you are doing this in the wrong way.
Everything should be described in the template, and a resource backed by an external service
should provide this functionality.

have a look at this: (it provides server resarting)
https://github.com/openstack/heat-templates/blob/master/cfn/WordPress_Single_Instance_With_IHA.template

We are working on using Ceilometer for alarms. Ceilometer consumes all the notifcations from the openstack projects.
We can then create alarm resources within the template (which sets up an alarm in ceilometer) then you need a policy resource
that will handle actioning the alarm.

 https://github.com/openstack/heat/blob/master/heat/engine/resources/instance.py#L39

I encourage you get involved in the upstream development, or to at least convey your use case so
we can build that functionality in.

Revision history for this message
Steven Hardy (shardy) said : 		#4

As Angus has outlined, several aspects of this may become easier when the ceilometer integration is completed. You can track this work via this BP:

https://blueprints.launchpad.net/heat/+spec/watch-ceilometer

Part of this work will be to implement a "webhook" style interface to allow notifications to heat on alarm state changes.

It seems like nova notifications should be something we can receive via the same/similar mechanism, or perhaps ceilometer will allow us to set alarms on specific nova notifications directly.

The other aspect of nova notifications which we would like to investigate is using notifications for state-changes (e.g BUILD to ACTIVE), to avoid the polling of nova instance state that we currently do:

https://github.com/openstack/heat/blob/master/heat/engine/resources/instance.py#L371

Please do come and get involved with the community so we can more fully understand your use-case! :)

Revision history for this message
Samuli Silvius (samuli-silvius) said : 		#5

Thanks for your replies so far!

For a short time solution I could try to integrate the whole functionality inside heat-engine as proposed. But this future solution sounds better as later we would also like to consume ceilometer meters concerning stack instances.

Our current use case is to get immediate info (nova notification) about instance stopped (e.g. killed with kill -9) and restart it if certain MetricName/Namespace defined in Alarm for that instance (having restarter policy also).

Is there already some ready code + examples templates about "webhook" way to get notifications to heat? Does that solution use ceilometer current (grizzly) api or does that need changes to ceilometer also?

This area heat-ceilometer and getting "fault management" and "performance management" related online information could be the area where to contribute if getting familiar enough to these plans.

Revision history for this message
Steven Hardy (shardy) said : 		#6

Hi,

The use case you describe is already supported, but not yet via nova notifications, see these templates (one as previously linked by Angus);

https://github.com/openstack/heat-templates/blob/master/cfn/WordPress_Single_Instance_With_HA.template
https://github.com/openstack/heat-templates/blob/master/cfn/WordPress_Single_Instance_With_IHA.template

The first template restarts the httpd service if it is not running, then rebuilds the instance if this fails more than a defined number of times.

The second template sends a heartbeat message from the instance, such that it is rebuilt if the heartbeat timeout is exceeded.

Clearly using nova notifications to do this will be better long-term, but this is the solution we have today (in grizzly).

For havana, as previously outlined, things are set to change considerably - the watch-ceilometer BP is tracking the integration with the new-for-havana ceilometer alarms functionality. This work includes the "webhook" mechanism described, it does not yet exist, we're still working out how to implement it.

I would suggest not pursuing the option of implementing this all in heat-engine, as we plan to remove the periodic/watchrule functionality when the ceilometer integration is completed.

IMO your effort is likely to be better spent ensuring that the new ceilometer alarms functionality is capable of alarming in the scenarios where you wish to trigger heat actions from, then when we complete the watch-ceilometer work you should have a solution that works for you in havana.

Revision history for this message
Samuli Silvius (samuli-silvius) said : 		#7

Hi,

I got it working like you suggested, putting my code running inside the heat-engine. Then there is no problem with passwords read from database with " db_api.user_creds_get(rs.stack.user_creds_id)".

But we will be working on to better solution (ceilometer integration) as suggested by you. Hope we could contribute at some point.