[Supervisor-checkins] r858 - superlance/trunk superlance/trunk/superlance supervisor/trunk supervisor/trunk/src/supervisor supervisor_manual/trunk supervisor_manual/trunk/chapters supervisor_manual/trunk/html
Mike Naberezny
mike at maintainable.com
Fri May 22 18:58:17 EDT 2009
Author: Mike Naberezny <mike at maintainable.com>
Date: Fri May 22 18:58:17 2009
New Revision: 858
Log:
- The console script ``memmon``, introduced in Supervisor 3.0a4, has
been moved to Superlance (http://pypi.python.org/pypi/superlance).
The Superlance package contains other useful monitoring tools designed
to run under Supervisor.
Added:
superlance/trunk/superlance/memmon.py
- copied unchanged from r856, /supervisor/trunk/src/supervisor/memmon.py
Removed:
supervisor/trunk/src/supervisor/memmon.py
supervisor_manual/trunk/chapters/memmon.xml
Modified:
superlance/trunk/CHANGES.txt
superlance/trunk/README.txt
superlance/trunk/setup.py
superlance/trunk/superlance/tests.py
supervisor/trunk/CHANGES.txt
supervisor/trunk/setup.py
supervisor_manual/trunk/html/HTML.manifest
supervisor_manual/trunk/manual.xml
Modified: superlance/trunk/CHANGES.txt
==============================================================================
--- superlance/trunk/CHANGES.txt (original)
+++ superlance/trunk/CHANGES.txt Fri May 22 18:58:17 2009
@@ -1,6 +1,11 @@
superlance Changelog
====================
+0.5 (2009-05-22)
+
+- Added ``memmon`` script, originally bundled with supervisor and
+ now moved to superlance.
+
0.4 (2009-02-11)
----------------
Modified: superlance/trunk/README.txt
==============================================================================
--- superlance/trunk/README.txt (original)
+++ superlance/trunk/README.txt Fri May 22 18:58:17 2009
@@ -15,4 +15,114 @@
``crashmail`` -- This script will email a user when a process enters
the EXITED state unexpectedly.
+``memmon`` -- See the description below.
+
+Memmon Overview
+---------------
+
+memmon is a supervisor "event listener" which may be subscribed to a
+concrete TICK_x event. When memmon receives a TICK_x event (TICK_60 is
+recommended, indicating activity every 60 seconds), memmon checks that a
+configurable list of programs (or all programs running under supervisor) are
+not exceeding a configurable about of memory (resident segment size, or RSS).
+If one or more of these processes is consuming more than the amount of memory
+that memmon believes it should, memmon will restart the process. memmon can be
+configured to send an email notification when it restarts a process.
+
+memmon is known to work on Linux and Mac OS X, but has not been tested on
+other operating systems (it relies on ps output and command-line switches).
+
+memmon is incapable of monitoring the process status of processes which are
+not supervisord child processes.
+
+Memmon Command
+--------------
+
+memmon is a "console script" installed when you install supervisor. Although
+memmon is an executable program, it isn't useful as a general-purpose script:
+it must be run as a supervisor event listener to do anything useful. memmon
+accepts the following options:
+
+ Option Argument(s) Description
+ ----------- --------------- ----------------------
+ -h N/A Show program help.
+ --help
+
+ -p name/size pair A name/size pair, e.g. "foo=1MB". The name
+ --program represents the supervisor program name that
+ you'd like memmon to monitor, the size represents
+ the number of bytes (suffix-multiplied using "KB",
+ "MB" or "GB") that should be considered "too much".
+ Multiple -p options can be provided to memmon to
+ signify that you'd like to monitor more than one
+ program. Programs can be specified as a "namespec",
+ to disambiguate same-named programs in different
+ groups, e.g. "foo:bar" represents the program "bar"
+ in the "foo" group.
+
+ -g name/size pair A groupname/size pair, e.g. "group=1MB". The name
+ --groupname represents the supervisor group name that you'd
+ like memmon to monitor, the size represents the
+ number of bytes (suffix-multiplied using "KB", "MB"
+ or "GB") that should be considered "too much".
+ Multiple -g options can be provided to memmon to
+ signify that you'd like to monitor more than one
+ group. If any process in this group exceeds the
+ maximum, it will be restarted.
+
+ -a size A size (suffix-multiplied using "KB", "MB" or "GB")
+ --any that should be considered "too much". If any program
+ running as a child of supervisor exceeds this maximum,
+ it will be restarted. E.g. 100MB.
+
+ -s command A command that will send mail if passed the email
+ --sendmail body (including the headers). Defaults to
+ _program "/usr/sbin/sendmail -t -i". Specifying this doesn't
+ cause memmon to send mail by itself (see the
+ -m/--email option).
+
+ -m email address An email address to which to send email when a process
+ --email is restarted. By default, memmon will not send any mail
+ unless an email address is specified.
+
+Configuring Memmon Into the Supervisor Config
+---------------------------------------------
+
+An [eventlistener:x] section must be placed in supervisord.conf in order for
+memmon to begin working. See the "Events" chapter in the Supervisor manual
+for more information about event listeners. The following examples assume that
+the memmon is on your system PATH.
+
+ memmon Example Configuration 1
+
+ This configuration causes memmon to restart any process which is a
+ child of supervisord consuming more than 200MB of RSS, and will send
+ mail to bob at example.com when it restarts a process using the default
+ sendmail command.
+
+ [eventlistener:memmon]
+ command=memmon -a 200MB -m bob at example.com
+ events=TICK_60
+
+ memmon Example Configuration 2
+
+ This configuration causes memmon to restart any process with the
+ supervisor program name "foo" consuming more than 200MB of RSS, and
+ will send mail to bob at example.com when it restarts a process using the
+ default sendmail command.
+
+ [eventlistener:memmon]
+ command=memmon -p foo=200MB -m bob at example.com
+ events=TICK_60
+
+ memmon Example Configuration 3
+
+ This configuration causes memmon to restart any process in the process
+ group "bar" consuming more than 200MB of RSS, and will send mail to
+ bob at example.com when it restarts a process using the default sendmail
+ command.
+
+ [eventlistener:memmon]
+ command=memmon -g bar=200MB -m bob at example.com
+ events=TICK_60
Modified: superlance/trunk/setup.py
==============================================================================
--- superlance/trunk/setup.py (original)
+++ superlance/trunk/setup.py Fri May 22 18:58:17 2009
@@ -16,7 +16,7 @@
CHANGES = ''
setup(name='superlance',
- version='0.4',
+ version='0.5',
description='superlance plugins for supervisord',
long_description=README + '\n\n' + CHANGES,
classifiers=[
@@ -48,6 +48,7 @@
[console_scripts]
httpok = superlance.httpok:main
crashmail = superlance.crashmail:main
+ memmon = superlance.memmon:main
"""
)
Modified: superlance/trunk/superlance/tests.py
==============================================================================
--- superlance/trunk/superlance/tests.py (original)
+++ superlance/trunk/superlance/tests.py Fri May 22 18:58:17 2009
@@ -301,6 +301,198 @@
self.failUnless(
'Process foo in group bar exited unexpectedly' in mail)
+
+class MemmonTests(unittest.TestCase):
+ def _getTargetClass(self):
+ from supervisor.memmon import Memmon
+ return Memmon
+
+ def _makeOne(self, *opts):
+ return self._getTargetClass()(*opts)
+
+ def _makeOnePopulated(self, programs, groups, any):
+ from supervisor.tests.base import DummyRPCServer
+ rpc = DummyRPCServer()
+ sendmail = 'cat - > /dev/null'
+ email = 'chrism at plope.com'
+ memmon = self._makeOne(programs, groups, any, sendmail, email, rpc)
+ memmon.stdin = StringIO()
+ memmon.stdout = StringIO()
+ memmon.stderr = StringIO()
+ memmon.pscommand = 'echo 22%s'
+ return memmon
+
+ def test_runforever_notatick(self):
+ programs = {'foo':0, 'bar':0, 'baz_01':0 }
+ groups = {}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:NOTATICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ self.assertEqual(memmon.stderr.getvalue(), '')
+
+ def test_runforever_tick_programs(self):
+ programs = {'foo':0, 'bar':0, 'baz_01':0 }
+ groups = {}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 8)
+ self.assertEqual(lines[0], 'Checking programs foo=0, bar=0, baz_01=0')
+ self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+ self.assertEqual(lines[2], 'Restarting foo:foo')
+ self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
+ self.assertEqual(lines[4], 'Restarting bar:bar')
+ self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
+ self.assertEqual(lines[6], 'Restarting baz:baz_01')
+ self.assertEqual(lines[7], '')
+ mailed = memmon.mailed.split('\n')
+ self.assertEqual(len(mailed), 4)
+ self.assertEqual(mailed[0], 'To: chrism at plope.com')
+ self.assertEqual(mailed[1],
+ 'Subject: memmon: process baz:baz_01 restarted')
+ self.assertEqual(mailed[2], '')
+ self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+ def test_runforever_tick_groups(self):
+ programs = {}
+ groups = {'foo':0}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 4)
+ self.assertEqual(lines[0], 'Checking groups foo=0')
+ self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+ self.assertEqual(lines[2], 'Restarting foo:foo')
+ self.assertEqual(lines[3], '')
+ mailed = memmon.mailed.split('\n')
+ self.assertEqual(len(mailed), 4)
+ self.assertEqual(mailed[0], 'To: chrism at plope.com')
+ self.assertEqual(mailed[1],
+ 'Subject: memmon: process foo:foo restarted')
+ self.assertEqual(mailed[2], '')
+ self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+ def test_runforever_tick_any(self):
+ programs = {}
+ groups = {}
+ any = 0
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 8)
+ self.assertEqual(lines[0], 'Checking any=0')
+ self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+ self.assertEqual(lines[2], 'Restarting foo:foo')
+ self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
+ self.assertEqual(lines[4], 'Restarting bar:bar')
+ self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
+ self.assertEqual(lines[6], 'Restarting baz:baz_01')
+ self.assertEqual(lines[7], '')
+ mailed = memmon.mailed.split('\n')
+ self.assertEqual(len(mailed), 4)
+
+ def test_runforever_tick_programs_and_groups(self):
+ programs = {'baz_01':0}
+ groups = {'foo':0}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 7)
+ self.assertEqual(lines[0], 'Checking programs baz_01=0')
+ self.assertEqual(lines[1], 'Checking groups foo=0')
+ self.assertEqual(lines[2], 'RSS of foo:foo is 2264064')
+ self.assertEqual(lines[3], 'Restarting foo:foo')
+ self.assertEqual(lines[4], 'RSS of baz:baz_01 is 2265088')
+ self.assertEqual(lines[5], 'Restarting baz:baz_01')
+ self.assertEqual(lines[6], '')
+ mailed = memmon.mailed.split('\n')
+ self.assertEqual(len(mailed), 4)
+ self.assertEqual(mailed[0], 'To: chrism at plope.com')
+ self.assertEqual(mailed[1],
+ 'Subject: memmon: process baz:baz_01 restarted')
+ self.assertEqual(mailed[2], '')
+ self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+ def test_runforever_tick_programs_norestart(self):
+ programs = {'foo': sys.maxint}
+ groups = {}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 3)
+ self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
+ self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+ self.assertEqual(lines[2], '')
+ self.assertEqual(memmon.mailed, False)
+
+ def test_stopprocess_fault_tick_programs_norestart(self):
+ programs = {'foo': sys.maxint}
+ groups = {}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ memmon.runforever(test=True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 3)
+ self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
+ self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+ self.assertEqual(lines[2], '')
+ self.assertEqual(memmon.mailed, False)
+
+ def test_stopprocess_fails_to_stop(self):
+ programs = {'BAD_NAME': 0}
+ groups = {}
+ any = None
+ memmon = self._makeOnePopulated(programs, groups, any)
+ memmon.stdin.write('eventname:TICK len:0\n')
+ memmon.stdin.seek(0)
+ from supervisor.process import ProcessStates
+ memmon.rpc.supervisor.all_process_info = [ {
+ 'name':'BAD_NAME',
+ 'group':'BAD_NAME',
+ 'pid':11,
+ 'state':ProcessStates.RUNNING,
+ 'statename':'RUNNING',
+ 'start':0,
+ 'stop':0,
+ 'spawnerr':'',
+ 'now':0,
+ 'description':'BAD_NAME description',
+ } ]
+ import xmlrpclib
+ self.assertRaises(xmlrpclib.Fault, memmon.runforever, True)
+ lines = memmon.stderr.getvalue().split('\n')
+ self.assertEqual(len(lines), 4)
+ self.assertEqual(lines[0], 'Checking programs BAD_NAME=%s' % 0)
+ self.assertEqual(lines[1], 'RSS of BAD_NAME:BAD_NAME is 2264064')
+ self.assertEqual(lines[2], 'Restarting BAD_NAME:BAD_NAME')
+ self.failUnless(lines[3].startswith('Failed'))
+ mailed = memmon.mailed.split('\n')
+ self.assertEqual(len(mailed), 4)
+ self.assertEqual(mailed[0], 'To: chrism at plope.com')
+ self.assertEqual(mailed[1],
+ 'Subject: memmon: failed to stop process BAD_NAME:BAD_NAME, exiting')
+ self.assertEqual(mailed[2], '')
+ self.failUnless(mailed[3].startswith('Failed'))
+
+
def make_connection(response, exc=None):
class TestConnection:
def __init__(self, hostport):
Modified: supervisor/trunk/CHANGES.txt
==============================================================================
--- supervisor/trunk/CHANGES.txt (original)
+++ supervisor/trunk/CHANGES.txt Fri May 22 18:58:17 2009
@@ -1,4 +1,9 @@
Next Release
+
+ - The console script ``memmon``, introduced in Supervisor 3.0a4, has
+ been moved to Superlance (http://pypi.python.org/pypi/superlance).
+ The Superlance package contains other useful monitoring tools designed
+ to run under Supervisor.
- Supervisorctl now correctly interprets all of the error codes that can
be returned when starting a process. Patch by Francesc Alted.
Modified: supervisor/trunk/setup.py
==============================================================================
--- supervisor/trunk/setup.py (original)
+++ supervisor/trunk/setup.py Fri May 22 18:58:17 2009
@@ -96,7 +96,6 @@
'supervisorctl = supervisor.supervisorctl:main',
'echo_supervisord_conf = supervisor.confecho:main',
'pidproxy = supervisor.pidproxy:main',
- 'memmon = supervisor.memmon:main',
],
},
)
Modified: supervisor_manual/trunk/html/HTML.manifest
==============================================================================
--- supervisor_manual/trunk/html/HTML.manifest (original)
+++ supervisor_manual/trunk/html/HTML.manifest Fri May 22 18:58:17 2009
@@ -7,7 +7,6 @@
./logging.html
./events.html
./extending_supervisors_xmlrpc_api.html
-./memmon.html
./upgrading_from_older_releases.html
./faq.html
./development.html
Modified: supervisor_manual/trunk/manual.xml
==============================================================================
--- supervisor_manual/trunk/manual.xml (original)
+++ supervisor_manual/trunk/manual.xml Fri May 22 18:58:17 2009
@@ -47,7 +47,6 @@
&chapters.logging;
&chapters.events;
&chapters.rpcinterfaces;
- &chapters.memmon;
&chapters.upgrading;
&chapters.faq;
&chapters.development;
More information about the Supervisor-checkins
mailing list