supervisor から solr を起動してプロセス死亡時に再起動するようにしたときのメモ (ざっくり)

supervisorの設定

systemのpythonに直接インストールする

$ sudo pip install supervisor
$ sudo mkdir /var/log/supervisor/
$ sudo mkdir /etc/supervisord.d/
$ sudo su - root -c "echo_supervisord_conf > /etc/supervisord.conf"
$ sudo etckeeper commit -m"Add supervisord.conf"

各設定ファイルを記述

supervisorのグローバル設定

• /etc/supervisord.conf

diff --git a/supervisord.conf b/supervisord.conf
index a29252e..b076add 100644
--- a/supervisord.conf
+++ b/supervisord.conf
@@ -13,11 +13,11 @@ file=/tmp/supervisor.sock   ; (the path to the socket file)
 ;password=12311 @@ file=/tmp; (default is no password (open server))
 [supervisord]
-logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
+logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
 logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
 logfile_backups=10           ; (num of main logfile rotation backups;default 10)
 loglevel=infops=10           ; (log level;default info; others: debug,warn,trace)
-pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
+pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
 nodaemon=falseun/supervisord.; (start in foreground if true;default false)
 minfds=1024lseun/supervisord.; (min. avail startup file descriptors;default 1024)
 minprocs=200seun/supervisord.; (min. avail process descriptors;default 200)
@@ -128,4 +128,4 @@ serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
 ; include files themselves.
 [include]
-;files = relative/directory/*.ini
+files = /etc/supervisord.d/*.ini

solrの設定

• /etc/supervisor.d/solr.ini

-;files = relative/directory/*.ini
+;files = /etc/supervisord.d/*.ini
diff --git a/supervisord.d/solr.ini b/supervisord.d/solr.ini
new file mode 100644
index 0000000..b8ca6de
--- /dev/null
+++ b/supervisord.d/solr.ini
@@ -0,0 +1,8 @@
+[program:solr]
+command=java -jar start.jar
+directory=/home/moqada/sandbox/apache-solr-3.6.0/example/
+stdout_logfile_maxbytes=50MB
+stderr_logfile_maxbytes=50MB
+stdout_logfile=/var/log/supervisor/solr.log
+stderr_logfile=/var/log/supervisor/solr_err.log
+auto_restart=true

起動確認

$ sudo supervisord
$ sudo supervisorctl status
solr                             RUNNING    pid 28985, uptime 0:05:41

topでもsolrとsupervisorが起動していることを確認

自動起動設定

supervisor自身の自動起動のためにupstartを導入

インストール

$ sudo yum install upstart

設定ファイル記述

/etc/init/supervisord.conf

diff --git a/init/supervisord.conf b/init/supervisord.conf
new file mode 100644
index 0000000..12bf7fa
--- /dev/null
+++ b/init/supervisord.conf
@@ -0,0 +1,7 @@
+description     "supervisor"
+
+start on runlevel[2345]
+stop on runlevel[!2345]
+
+respawn
+exec /usr/bin/supervisord -n

起動

$ sudo init start supervisord

killして再起動されるか確認

操作諸々

起動

$ sudo supervisorctl start solr

停止

$ sudo supervisorctl stop solr

再起動

sudo supervisorctl restart solr

設定再読み込み

$ sudo supervisorctl reread solr

最終的にsolrはmulticore起動するようにsolr.iniのcommandを変更

[program:solr]
command=java -Dsolr.solr.home=multicore -jar start.jar
directory=/home/moqada/sandbox/apache-solr-3.6.0/example/
stdout_logfile_maxbytes=50MB
stderr_logfile_maxbytes=50MB
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s_err.log
auto_restart=true

プロセス死亡時のメール通知設定

http://blog.hakutoitoi.com/?p=396

$ sudo pip install superlance

設定ファイルを追加
なんとなくそれように分割

/etc/supervisord.d/crashmail.ini

diff --git a/supervisord.d/crashmail.ini b/supervisord.d/crashmail.ini
new file mode 100644
index 0000000..743d9ec
--- /dev/null
+++ b/supervisord.d/crashmail.ini
@@ -0,0 +1,3 @@
+[eventlistener:crashmail]
+command=/usr/bin/crashmail -p solr -m example@example.com
+events=PROCESS_STATE_EXITED

起動、確認

$ sudo supervisorctl update
$ sudo supervisorctl status
crashmail                        RUNNING    pid 29422, uptime 0:03:54
solr                             RUNNING    pid 29429, uptime 0:03:23

killして確認してみる

メールが届いたらOK