Priority appears unused in storage-schemas.conf

This patch appears to make priority work.

diff --git a/carbon-0.9.7/lib/carbon/storage.py b/carbon-0.9.7/lib/carbon/storage.py
index 3110f10..79df559 100644
--- a/carbon-0.9.7/lib/carbon/storage.py
+++ b/carbon-0.9.7/lib/carbon/storage.py
@@ -15,6 +15,7 @@ limitations under the License."""
 import os, re
 from os.path import join, exists
 from carbon.conf import OrderedConfigParser, Settings, settings
+from operator import attrgetter

 try:
   import cPickle as pickle
@@ -77,30 +78,33 @@ class Schema:

 class DefaultSchema(Schema):
- def __init__(self, name, archives):
+ def __init__(self, name, archives, priority):
     self.name = name
     self.archives = archives
+ self.priority = priority

   def test(self, metric):
     return True

 class PatternSchema(Schema):
- def __init__(self, name, pattern, archives):
+ def __init__(self, name, pattern, archives, priority):
     self.name = name
     self.pattern = pattern
     self.regex = re.compile(pattern)
     self.archives = archives
+ self.priority = priority

   def test(self, metric):
     return self.regex.search(metric)

  class ListSchema(Schema):
- def __init__(self, name, listName, archives):
+ def __init__(self, name, listName, archives, priority):
     self.name = name
     self.listName = listName
     self.archives = archives
+ self.priority = priority
     self.path = join(WHITELISTS_DIR, listName)

     if exists(self.path):
@@ -154,6 +158,7 @@ def loadStorageSchemas():
     matchAll = options.get('match-all')
     pattern = options.get('pattern')
     listName = options.get('list')
+ priority = options.get('priority')

     retentions = options['retentions'].split(',')
     archives = [ Archive.fromString(s) for s in retentions ]
@@ -162,7 +167,7 @@ def loadStorageSchemas():
       mySchema = DefaultSchema(section, archives)

     elif pattern:
- mySchema = PatternSchema(section, pattern, archives)
+ mySchema = PatternSchema(section, pattern, archives, priority)

     elif listName:
       mySchema = ListSchema(section, listName, archives)
@@ -173,8 +178,10 @@ def loadStorageSchemas():
     schemaList.append( mySchema )

   schemaList.append( defaultSchema )
+ schemaList = sorted(schemaList,key=attrgetter('priority'),reverse=True)
+
   return schemaList

 defaultArchive = Archive(60, 60 * 24 * 7) #default retention for unclassified data (7 days of minutely data)
-defaultSchema = DefaultSchema('default', [defaultArchive])
+defaultSchema = DefaultSchema('default', [defaultArchive], 0)

A.) You're correct, priority is no longer used. The rules are tested in order, and the first match is used. Since everything_1min_1day matches anything, all metrics will use it.

B) the retention is in the following format:

seconds_per_point:number_of_points_to_store

So 60 seconds = 1 minute, 1440 minutes = 1 day.

There's a new way to set these with min, h, d (minute hour day) suffixes as well.

-Nick

On Feb 22, 2011, at 4:22 PM, Kindjal <email address hidden> wrote:

> New question #146483 on Graphite:
> https://answers.launchpad.net/graphite/+question/146483
>
> I couldn't figure out why my storage schema was not being used. It appears that "priority" is completely unused.
>
> # This is from .example.conf
> [everything_1min_1day]
> priority = 90 # I tried using 90 and 110
> pattern = .*
> retentions = 60:1440
>
> # These are the schemas I want to actually use
> [foo]
> priority = 100
> pattern = ^foo\.
> retentions = 1800:8640
>
> [bar]
> priority = 100
> pattern = ^bar\.
> retentions = 60:43200,900:350400
>
> Neither foo nor bar were ever used. Only the default. Regardless of priority. I grepped through the source and don't find "priority" used anywhere. Am I missing something?
>
> Also, in the above schemas, it looks like the default, which claims to save 24 hours of data, really only saves 24 minutes of data. I wonder if there's a missing * 60 multiplier someplace.
>
>
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp

"A.) You're correct, priority is no longer used. The rules are tested in order, and the first match is used. Since everything_1min_1day matches anything, all metrics will use it."

Ok, thanks. I guess I would suggest describing that fact in the Quickstart Guide and other documentation.

"B) the retention is in the following format:

seconds_per_point:number_of_points_to_store

So 60 seconds = 1 minute, 1440 minutes = 1 day."

Yes that's what I thought would be true, but experimentation showed that that retention policy was saving 24 minutes of data, not 24 hours of data.

If I start with an empty configuration and use the 60:1440 retention policy, then fill up the DB with sample data, it is always limited to 24 minutes.

Which version of graphite are you running?

-Nick

On Feb 23, 2011, at 10:58 AM, Kindjal <email address hidden> wrote:

> Question #146483 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/146483
>
> Kindjal posted a new comment:
> "A.) You're correct, priority is no longer used. The rules are tested in
> order, and the first match is used. Since everything_1min_1day matches
> anything, all metrics will use it."
>
> Ok, thanks. I guess I would suggest describing that fact in the
> Quickstart Guide and other documentation.
>
> "B) the retention is in the following format:
>
> seconds_per_point:number_of_points_to_store
>
> So 60 seconds = 1 minute, 1440 minutes = 1 day."
>
> Yes that's what I thought would be true, but experimentation showed that
> that retention policy was saving 24 minutes of data, not 24 hours of
> data.
>
> If I start with an empty configuration and use the 60:1440 retention
> policy, then fill up the DB with sample data, it is always limited to 24
> minutes.
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp

graphite-web 0.9.7c with 0.9.7 carbon and whisper.

I'm seeing the same problem with retention on 0.9.7c/0.9.7. I have the following in storage-schemas.conf:

[metrics]
priority = 110
pattern = ^metrics\..*
retentions = 10:2160,60:10080,600:262974

I believe I should be getting 6 hours of retention at 10 second intervals, but I only get 36 minutes. whisper-info.py shows only 216 points in Archive 0 after nearly 24 hours of collecting, instead of 2160:

maxRetention: 262800
xFilesFactor: 0.5
fileSize: 9916

Archive 0
retention: 2160
secondsPerPoint: 10
points: 216
size: 2592
offset: 52

Archive 1
retention: 10080
secondsPerPoint: 60
points: 168
size: 2016
offset: 2644

Archive 2
retention: 262800
secondsPerPoint: 600
points: 438
size: 5256
offset: 4660

Update: this revision http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/346 fixes the retention problem.