Graphite

Webapp not pulling data from remote backends

Asked by ZachG

Hi! I'm using the graphite 0.9.5 stack and have the following architecture:

Machine 1 Running Carbon Relay + Webapp.
Machine 2 Running Carbon Cache

I have confirmed that everything is setup correctly and that data sent to the relay does properly get written to disk by the cache server. I then went into the webapp folder on the relay server and edited local_settings.,py too have:

CARBONLINK_HOSTS = ["cache1:7002"]
CLUSTER_SERVERS = ["cache1:7002"]

I then try and expand the graphite folder on the webapp and it doesn't return the proper folders (not even the carbon/agents or carbon/relay folders). In the carbon cache logs I do see:

16/02/2010 19:38:47 :: [listener] CacheQueryHandler connection with XXXX:50105 established
16/02/2010 19:38:47 :: [listener] CacheQueryHandler connection with XXXX:50105 closed cleanly

I also did a tcpdump of port 7002 and saw the incomming GET request, but no response. Any idea why the cache isn't returning data to the webserver? I'm using twisted 0.9 (from "easy_install twisted").

Thanks for any help you can provide!

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
chrismd
Solved:
Last query:
Last reply:
Revision history for this message
Best chrismd (chrismd) said : 		#1

Hi sorry for the delay in getting back to you, I took a few days off of working on graphite :)

So I am planning to focus on documentation in March, I should get some basic stuff about how to setup the cluster but here's a quick run down of the problem.

The webapp serves requests using local whisper data files. Carbon-cache receives data and writes it to local whisper data files. This means carbon-cache and the webapp should be on the same machine. Carbon-relay is just an application-level load balancer for when you have multiple servers running carbon-cache.

If you want to use 2 servers in a cluster you should have both run carbon-cache, both run the webapp, and at least one running carbon-relay (both for high availability) with a relay-rules.conf file that specifies when to send traffic to whichever carbon-cache instance.

Both webapp instances need to have CLUSTER_SERVERS = ["localhost", "theotherguy"] but if you specify a port it must be the web server port. This setting is used to talk to other webapp instances via HTTP.

Also both webapps need to have the default CARBONLINK_HOSTS setting, just leave it unchanged.

Hopefully this will all make more sense when I finish the docs.

Revision history for this message
ZachG (zgold550) said : 		#2

Chris,

I'm still a bit confused. My config is now:

Relay Machine:
CARBONLINK_HOSTS = ["127.0.0.1:7002"]
CLUSTER_SERVERS = ["relay1.graphite.company.com", "localhost"]

Carbon Machine:
CARBONLINK_HOSTS = ["127.0.0.1:7002"]
CLUSTER_SERVERS = ["carbon1.graphite.company.com", "localhost"]

Both machines are running the webapp. On the carbon machine the webapp shows the local data. On the relay machine it gives me nothing. (For now I get a complete view via the carbon machine. I am, however, intending on adding more carbon backends.)

I got the impression from the federated storage description on the website that the webapp should be able to pull data from a different machine. In the situation you describe above, with two different carbon backends and a webapp on each, shouldn't either webapp be able to pull data from the other to get a complete view of the data? If so, why should a webapp thats on a relay machine not be able to show data from a webapp on a carbon machine (or maybe it can and my settings are still wrong? Perhaps i need to specify a different port? All webapps are running via apache on port 80 )

Thanks again for any help you can provide!
-Zach

Revision history for this message
ZachG (zgold550) said : 		#3

Chris,

I switched the config in my comment above. The carbon config points to relay1 and relay machine points to carbon1. On the machines themselves it is correct.

-Zach

Revision history for this message
ZachG (zgold550) said : 		#4

Actually, I am mistaken. It was backwards in the machine configuration as well. All is working now :)

Revision history for this message
ZachG (zgold550) said : 		#5

Thanks chrismd, that solved my question.

Revision history for this message
JOhn Chang (jchang) said : 		#6

Chris,
I can get this working, but not as described here. If I have a "master" graphite that has this config:
CLUSTER_SERVERS=['server1:port', 'server2:port']

and then for server1 and server2, I configure:
CLUSTER_SERVERS=[]

I hit the master server in the browser, and all works out great.

If I do as you suggest above, it seems to hang. I try to browse through the tree, and it just hangs as long as server1 or server2 is up. I'm wondering if I get caught in a loop where master and the server1/2 just call back and forth to each other.

Functionally, I am happy with what I have, but for deployment simplification, it would be nice if I could get it working as you describe above.

Any help you can afford would be kindly appreciated.