zenticket performs the following procedures each cycle:
The ticket create script performs the following:
The zenticket daemon is configured via the zenticket.conf file. This file is editable by logging in to the Zenoss server via SSH, becoming the zenoss user, and then editing $ZENHOME/etc/zenticket.conf. It is also editable via the Zenoss user interface by logging in and navigating to Settings → Daemons then selecting “view config” for zenticket, then selecting “edit this configuration”.
The zenticket.conf file looks like the following:
# Examples: # # Single client Zenoss server: # # [CUSTOMER NAME] # id:cust-00000 # queue:Front Line # group1:None # group2:/Network # group3:/Security # group4:/Server # group5:/Up-Down # # Multi-client Zenoss server: # # [CUSTOMER NAME] # id:cust-00000 # queue:Front Line # customer=CUSTOMER NAME # group1:/%(customer)s # group2:/%(customer)s/Network # group3:/%(customer)s/Security # group4:/%(customer)s/Server # group5:/%(customer)s/Up-Down [GENERAL] ticketscript:/home/zenoss/zenticket/create_ticket.pl cycletime:30 [LAB] id:nnl-00000 queue:Front Line group1:None group2:/Network group3:/Security group4:/Server group5:/Up-Down
As you can see in the examples which are commented out in the config, the config file can be used for Zenoss servers which are dedicated to a single client, as well as Zenoss servers which monitor multiple clients. The path to the ticket create script, the cycle time of the daemon, customer id, queue, and customer groups can all be defined in the zenticket.conf file.
In this case the path to the ticket script is set to /home/zenoss/zenticket/create_ticket.pl, the cycle time is set to 30 seconds, and the client LAB has been given a customer id of nnl-00000 and a queue of Front Line. There are also groups defined for this client. If an event comes in for a device in any of the device groups listed the ticket will be generated in the Front Line queue with a customer id of nnl-00000.
After the config file for the daemon is edited the daemon must be restarted so that it picks up on the new configuration. This can be done in two ways. The first is by logging in to the Zenoss server via SSH, becoming the zenoss user, and executing “zenticket restart”. The second is by logging in to the Zenoss user interface, navigating to Settings → Daemons and clicking on “Restart” for zenticket.
The zenticket daemon currently sends log info to $ZENHOME/log/zenticket.log. This log file is either viewable via SSH or via the user interface by logging in and navigating to Settings → Daemons and clicking “view log” for zenticket.
At the moment logging is fairly limited, but I do intend to implement more detailed logging in a future version of the daemon.
Example log file output:
2009-11-18 16:45:33 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:46:04 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:46:35 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:47:10 INFO zen.zenticket: ticket create script ran 6 times 2009-11-18 16:47:55 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:48:26 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:48:58 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:49:29 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:50:00 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:50:32 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:51:03 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:51:34 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:52:06 INFO zen.zenticket: ticket create script ran 2 times 2009-11-18 16:53:19 INFO zen.zenticket: Deleting PID file /usr/local/zenoss/zenoss/var/zenticket-localhost.pid ... 2009-11-18 16:53:19 INFO zen.zenticket: zenticket shutting down 2009-11-18 16:53:43 INFO zen.zenticket: Starting zenticket 2009-11-18 16:54:19 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:54:50 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:55:22 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:55:53 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:56:24 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:56:56 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:57:27 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:57:58 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:58:29 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:59:01 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 16:59:32 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 17:00:09 INFO zen.zenticket: ticket create script ran 6 times 2009-11-18 17:00:46 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 17:01:30 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 17:02:02 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 17:02:33 INFO zen.zenticket: ticket create script ran 1 time 2009-11-18 17:03:05 INFO zen.zenticket: ticket create script ran 1 time
Here is the current code for the zenticket daemon (it is written in python):
#!/usr/bin/env python
# Perform initial imports.
from daemon import Daemon
import os, sys
# Discover paths to files.
pidfile = os.path.join(os.environ['ZENHOME'], 'var/zenticket-localhost.pid')
zenconfpath = os.path.join(os.environ['ZENHOME'], 'etc/zenticket.conf')
logfile = os.path.join(os.environ['ZENHOME'], 'log/zenticket.log')
# Configure logging.
import logging
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(levelname)s zen.zenticket: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
filename=logfile,
filemode='a')
# Daemon code space begins here.
class MyDaemon(Daemon):
def run(self):
# Perform Zenoss specific imports.
import Globals
from Products.ZenUtils.ZenScriptBase import ZenScriptBase
from transaction import commit
dmd = ZenScriptBase(connect=True).dmd
from Products.ZenUtils import Time
# Perform other necessary imports.
from subprocess import call
from MySQLdb import OperationalError
import time, socket, re, subprocess, ConfigParser, datetime
# Read in config file.
config = ConfigParser.ConfigParser()
config.read([zenconfpath])
# Gather general config file options in to variables.
ticketscript = config.get("GENERAL", "ticketscript")
cycletime = config.get("GENERAL", "cycletime")
autocleartime = config.get("GENERAL", "autocleartime")
# Create events dictionary
events = {}
# Configure logging within daemon code space.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(levelname)s zen.zenticket: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
filename=logfile,
filemode='a')
sys.stderr = open(logfile, 'a')
# Daemon cycle begins here.
while True:
# Configure initial variables for daemon.
ticketscreated = 0
evt = 0
delevents = []
# Generate list of events to remove from the event dictionary
# (if they are no longer present in the Zenoss event console).
for k, v in events.iteritems():
eventmatch = 0
for e in dmd.ZenEventManager.getEventList([], "", "lastTime ASC, firstTime ASC"):
if re.match(k, e.evid):
eventmatch = 1
if eventmatch == 0:
delevents.append(k)
# Go through list of events to remove and remove events from the events dictionary.
for e in delevents:
del events[e]
# Ticket creation cycle begins here.
for e in dmd.ZenEventManager.getEventList([], "", "lastTime ASC, firstTime ASC"):
# Define initial variables for ticket creation cycle.
delevent = False
evt = None
create = 0
# Check if the event count has increased since the last cycle (for existing events).
# If it has increased, trigger ticket create script and increase count in event dictionary.
if e.evid in events:
if e.count > events[e.evid]:
create = 1
events[e.evid] = e.count
evt = dmd.ZenEventManager.getEventDetailFromStatusOrHistory(e.evid)
else:
# If count has not increased, check if event is acknowledged.
# If event is not acknowledged, trigger ticket create script and set count in event dictionary.
# If event is acknowledged, simply set count in event dictionary.
evt = dmd.ZenEventManager.getEventDetailFromStatusOrHistory(e.evid)
if evt.eventState == 0:
create = 1
events[e.evid] = e.count
else:
events[e.evid] = e.count
if evt:
# If triggered, ticket create routine begins here.
if create == 1:
# Configure initial variables for ticket create routine.
ticket = None
p = None
groupmatch = 0
# Check that event does not match any filter rules (should not generate tickets for certain events).
if not re.match("Command timed out on device", evt.message) and evt.prodState == 1000 and evt.severity >= 3 and \
evt.summary != "Unknown" and not re.match("/Discovered", evt.DeviceClass):
# Check config file for matching device group.
for s in config.sections():
if not re.match('GENERAL', s):
groups = evt.DeviceGroups
groups = groups.split('|')
for g in groups:
if not re.match('^$', str(g)):
if str(config.get(s, "group1")) == str(g) or str(config.get(s, "group2")) == str(g) or \
str(config.get(s, "group3")) == str(g) or str(config.get(s, "group4")) == str(g) or \
str(config.get(s, "group5")) == str(g):
groupmatch = 1
# Grab customer id and queue information from the config file.
custid = config.get(s, "id")
queue = config.get(s, "queue")
# If a matching group is found in the config file, run the ticket create script.
if groupmatch == 1:
# Make sure the script exists before attempting to run it.
if os.path.exists(str(ticketscript)):
# Run the ticket create script (while passing necessary arguments to it).
p = subprocess.Popen([str(ticketscript), '-customer', str(custid), '-device',
str(evt.device), '-deviceIP', str(evt.ipAddress), '-collector', socket.getfqdn(),
'-first', str(evt.firstTime), '-last', str(evt.lastTime), '-count', str(evt.count),
'-summary', str(evt.summary), '-noteTitle', 'System Monitor Error', '-note',
str(evt.message), '-severity', str(evt.severity), '-group', str(evt.DeviceGroups),
'-impact', str(evt.DevicePriority), '-component', str(evt.component),
'-queue', str(queue)], stdout=subprocess.PIPE)
else:
# Write a warning to the log file if a device does not belong to any groups.
if evt.device != socket.getfqdn():
logging.warning("Device %s is not in a device group" % (evt.device))
# Check to make sure ticket creation was successful based on ticket number returned by script.
if p:
try:
ticket = int(p.stdout.read())
# If ticket create script passes a string as output, set ticket to 0 to indicate a failure.
# Write a warning to the log file as well.
except ValueError:
logging.warning("The ticket create script passed a string to zenticket")
ticket = 0
pass
else:
ticket = 0
# If ticket was successfully created, increment the count of ticketscreated.
# Also, acknowledge the event in Zenoss.
if ticket > 0:
ticketscreated = ticketscreated + 1
eventli = [evt.evid]
delevent = False
if evt.eventState == 0:
try:
dmd.ZenEventManager.manage_setEventStates(1, eventli)
# Ignore certain errors thrown by MySQL and Zenoss.
except OperationalError, e:
if e[0] == 1205:
pass
elif e[0] == 1213:
pass
elif e[0] == 1422:
pass
elif e[0] == 1206:
pass
elif e[0] == 2002:
pass
else:
raise
except ZenEventNotFound:
pass
else:
# If event did not match filters for ticket creation, move it to history.
# Only tickets that do not match the filters below will be automatically moved to history.
if not re.match('/Status/Ping', evt.eventClass) and not re.match('SNMP agent down', evt.summary) \
and not evt.severity == 1 and not re.match('interface operationally down', evt.summary) \
and not re.match('threshold of', evt.summary):
delevent = True
# Move events that exceed the configured autocleartime setting to history.
# This routine also deletes any events that were marked to be moved to history in previous routines.
evt = dmd.ZenEventManager.getEventDetailFromStatusOrHistory(e.evid)
pattern = '%Y/%m/%d %H:%M:%S'
epoch = int(time.mktime(time.strptime(evt.lastTime.split('.')[0], pattern)))
if time.time() - epoch > int(autocleartime) or delevent == True:
try:
dmd.ZenEventManager.manage_deleteEvents(evt.evid)
# Ignore certain errors thrown by MySQL and Zenoss.
except OperationalError, e:
if e[0] == 1205:
pass
elif e[0] == 1213:
pass
elif e[0] == 1422:
pass
elif e[0] == 1206:
pass
elif e[0] == 2002:
pass
else:
raise
except ZenEventNotFound:
pass
# Write activity summary to log file.
if ticketscreated > 0:
if ticketscreated == 1:
logging.info('Ticket create script ran %s time', ticketscreated)
else:
logging.info('Ticket create script ran %s times', ticketscreated)
# Sleep for the amount of seconds configured in the cycletime setting before starting the next cycle.
time.sleep(int(cycletime))
# Daemon runtime options are defined here.
if __name__ == "__main__":
daemon = MyDaemon(pidfile)
# Grab the option.
if len(sys.argv) == 2:
# Option to start the daemon.
if 'start' == sys.argv[1]:
if os.path.exists(zenconfpath):
logging.info('Starting zenticket')
daemon.start()
else:
print '%s is missing, aborting start.' % (zenconfpath)
# Option to stop the daemon.
elif 'stop' == sys.argv[1]:
if os.path.exists(pidfile):
logging.info('Deleting PID file %s ...', pidfile)
logging.info('zenticket shutting down')
print 'stopping...'
daemon.stop()
# Option to restart the daemon.
elif 'restart' == sys.argv[1]:
if os.path.exists(pidfile):
logging.info('Deleting PID file %s ...', pidfile)
logging.info('zenticket shutting down')
print 'stopping...'
daemon.stop()
if os.path.exists(zenconfpath):
logging.info('Starting zenticket')
daemon.start()
else:
print '%s is missing, aborting start.' % (zenconfpath)
# Option to get daemon status.
elif 'status' == sys.argv[1]:
try:
pf = file(daemon.pidfile,'r')
pid = int(pf.read().strip())
pf.close()
except IOError:
pid = None
# Function to check for the existence of a unix pid.
def check_pid(pid):
try:
os.kill(pid, 0)
except OSError:
return False
else:
return True
if not pid:
print 'not running'
else:
if check_pid(pid) == True:
print 'program running; pid=%s' % (pid)
else:
print 'not running'
# Option to generate XML options to be displayed when clicking on
# "edit config" in the Daemons section of the Zenoss UI.
elif 'genxmlconfigs' == sys.argv[1]:
print "\
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\
<configuration id=\"zenticket\" >\n\
<option id=\"\" type=\"string\" default=\"\" target=\"\" help=\"Edit%20the%20config%20\
file%20by%20navigating%20to%20view%20config%20->%20edit%20this%20\
configuration%20from%20the%20daemons%20page.\" />\n\
<option id=\" \" type=\"string\" default=\"\" target=\"\" help=\"Do%20not%20\
click%20save%20on%20this%20page%20as%20it%20will%20clear%20the%20config%20file.\" />\n\
<option id=\" \" type=\"string\" default=\"\" target=\"\" help=\"If%20the%20\
config%20does%20get%20cleared%20it%20can%20be%20restored%20from%20zenticket.conf.bak.\" />\n\
</configuration>"
else:
# Print valid options if invalid option is specified.
print "usage: zenticket start|stop|restart|status|genxmlconfigs"
sys.exit(2)
sys.exit(0)
else:
# Print valid options if invalid option is specified.
print "usage: zenticket start|stop|restart|status|genxmlconfigs"
sys.exit(2)
Here is the daemon script that the zenticket script uses to daemonize itself:
#!/usr/bin/env python
"""
Source: http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
Author: Sander Marechal
"""
import sys, os, time, atexit, signal
from signal import SIGTERM
class Daemon:
"""
A generic daemon class.
Usage: subclass the Daemon class and override the run() method
"""
def __init__(self, pidfile, stdin='/dev/null', stdout='/dev/null', stderr='/dev/null'):
self.stdin = stdin
self.stdout = stdout
self.stderr = stderr
self.pidfile = pidfile
def daemonize(self):
"""
do the UNIX double-fork magic, see Stevens' "Advanced
Programming in the UNIX Environment" for details (ISBN 0201563177)
http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
"""
try:
pid = os.fork()
if pid > 0:
# exit first parent
sys.exit(0)
except OSError, e:
sys.stderr.write("fork #1 failed: %d (%s)\n" % (e.errno, e.strerror))
sys.exit(1)
# decouple from parent environment
os.chdir("/")
os.setsid()
os.umask(0)
# do second fork
try:
pid = os.fork()
if pid > 0:
# exit from second parent
sys.exit(0)
except OSError, e:
sys.stderr.write("fork #2 failed: %d (%s)\n" % (e.errno, e.strerror))
sys.exit(1)
# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
si = file(self.stdin, 'r')
so = file(self.stdout, 'a+')
se = file(self.stderr, 'a+', 0)
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
# write pidfile
atexit.register(self.delpid)
pid = str(os.getpid())
file(self.pidfile,'w+').write("%s\n" % pid)
def delpid(self):
os.remove(self.pidfile)
def start(self):
"""
Start the daemon
"""
def check_pid(pid):
""" Check For the existence of a unix pid. """
try:
os.kill(pid, 0)
except OSError:
return False
else:
return True
# Check for a pidfile to see if the daemon already runs
try:
pf = file(self.pidfile,'r')
pid = int(pf.read().strip())
pf.close()
except IOError:
pid = None
if pid:
if check_pid(pid) == True:
message = "is already running\n"
sys.stderr.write(message)
sys.exit(1)
else:
pf = file(self.pidfile,'r')
pid = int(pf.read().strip())
pf.close()
# Start the daemon
message = "starting...\n"
sys.stderr.write(message)
self.daemonize()
self.run()
def stop(self):
"""
Stop the daemon
"""
# Get the pid from the pidfile
try:
pf = file(self.pidfile,'r')
pid = int(pf.read().strip())
pf.close()
except IOError:
pid = None
if not pid:
message = "already stopped\n"
sys.stderr.write(message)
return # not an error in a restart
# Try killing the daemon process
try:
while 1:
os.kill(pid, SIGTERM)
time.sleep(0.1)
except OSError, err:
err = str(err)
if err.find("No such process") > 0:
if os.path.exists(self.pidfile):
os.remove(self.pidfile)
else:
print str(err)
sys.exit(1)
def restart(self):
"""
Restart the daemon
"""
self.stop()
self.start()
def run(self):
"""
You should override this method when you subclass Daemon. It will be called after the process has been
daemonized by start() or restart().
"""