| « Programming using my LAN step one PhpMyAdmin | Automating some handy MythTV scripts » |
Zabbix + ssh tunnel with AutoSSH
2007/12/10
Zabbix + ssh tunnel with AutoSSH
I've spent a fair amount of time setting up a Zabbix system monitoring system to monitor the functioning of the LAN computers - Gentoo and Windows. Other than Zabbix 1.1.7 being a little tedious to set up, including the various add-on scanners I've configured, its fairly easy.
However, monitoring computers outside the LAN has been tricky, as I want to do it securely. Zabbix sends its info (at least in this version) through clear text, and I'd like to have in encrypted.
After several attempts at configuring AutoSSH (over several months :roll: ) I made some progress - first this article had some handy steps for testing:
namely:
Time to forward a port
a)add the port forward feature to the ssh command: '-L <local port>:system:<remote port>'
(b)'-L 1234:localhost:6667' would be my local system listening on port 1234 and sending requests to port 6667 of the remote's local loopback address (127.0.0.1)
(c)if forwarding is enabled on the remote system the you could use the remote system as a stepping stone to another.
(d)the new command would be:
(e)ssh -i <user's home dir>/.ssh/id_rsa -L <local port>:system:<remote port> <remote user>@<ip address>
(f)ie: 'ssh -i /root/.ssh/id_rsa -L 1234:localhost:6667 zxcv@207.46.19.30
(g)verify it works
(g)1.connect to the remote system
(g)2.set your irc client to connect locally on port 1234 which should forward the request to the remote system.
So, following these steps (I had already set up the RSA keys) I ended up with this command:
ssh -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 xxx.xxx.xx.xx
Finally, the Zabbix server process was getting results from the server outside the LAN:
With this config having been added to the '/etc/zabbix/zabbix_agentd.conf' file:
# List of comma delimited IP addresses (or hostnames) of ZABBIX servers.
# No spaces allowed. First entry is used for sending active checks.
# Note that hostnames must resolve hostname-[gt]IP address and
# IP address-[gt]hostname.
Server=127.0.0.1
# Server port for sending active checks
ServerPort=20050
# Unique hostname. Required for active checks.
Hostname=co-loserver
# Listen port. Default is 10050
ListenPort=10050
# IP address to bind agent
# If missing, bind to all available IPs
ListenIP=127.0.0.1
# Disable active checks. The agent will work in passive mode listening server.
DisableActive=1I disabled active checks, mostly as a first stage, but I also suspected there might be issues trying to get the active checks back thorugh the firewall for the LAN.
The next step in that article, about setting up AutoSSH, didn't work for me. The logs were showing results like:
rity ~ # tail -n2000 /var/log/messages | grep autossh
Dec 9 22:33:06 rity autossh[27800]: port set to 0, monitoring disabled
Dec 9 22:33:06 rity autossh[27801]: checking for grace period, tries = 0
Dec 9 22:33:06 rity autossh[27801]: starting ssh (count 1)
Dec 9 22:33:06 rity autossh[27802]: execing /usr/bin/ssh
Dec 9 22:33:06 rity autossh[27801]: ssh child pid is 27802
Dec 9 22:33:06 rity autossh[27801]: check on child 27802
Dec 9 22:33:06 rity autossh[27801]: set alarm for 600 secs
Dec 9 22:33:08 rity autossh[27801]: check on child 27802
Dec 9 22:33:08 rity autossh[27801]: ssh exited prematurely with status 0; autossh exiting
Dec 9 22:33:08 rity autossh[27801]: expired child, returning 2Eventually I found this entry (actually I had seen it before several times, but there was a new entry which made it all work):
http://gentoo-wiki.com/HOWTO_autossh
So, the command ends up looking like:
AUTOSSH_DEBUG=1 autossh -M0 -q -f -N -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 zabbix@xxx.xxx.xx.xx
I used the debug environment variable to assist in debugging any issues. I couldn't see any, finally (I've skipped over several false starts).
Here's what the log showed when AutoSSH was working properly:
Dec 9 23:45:22 rity autossh[16217]: port set to 0, monitoring disabled
Dec 9 23:45:22 rity autossh[16218]: checking for grace period, tries = 0
Dec 9 23:45:22 rity autossh[16218]: starting ssh (count 1)
Dec 9 23:45:22 rity autossh[16218]: ssh child pid is 16219
Dec 9 23:45:22 rity autossh[16218]: check on child 16219
Dec 9 23:45:22 rity autossh[16218]: set alarm for 600 secs
Dec 9 23:45:22 rity autossh[16219]: execing /usr/bin/ssh
Dec 9 23:55:22 rity autossh[16218]: check on child 16219
Dec 9 23:55:22 rity autossh[16218]: set alarm for 600 secsI used this to look at what ports Zabbix was using before the ssh tunnel was set up:
SP29 ~ # netstat -anp | grep zabbix
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:10050 0.0.0.0:* LISTEN 3999/zabbix_agentdHere's what it looked like after the AutoSSH tunnel was going:
SP29 ~ # netstat -anp | grep zabbix
tcp 0 0 127.0.0.1:10050 0.0.0.0:* LISTEN 8242/zabbix_agentd
tcp 0 0 64.251.22.129:22 24.84.203.91:3109 ESTABLISHED 14684/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 8748459 14686/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 8748457 14684/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 8748456 14686/sshd: zabbixStill however the Zabbix server was NOT receiving data over the AutoSSH tunnel. Very disappointing. Still I had managed to get an SSH tunnel to work, from the command line. There must be a way.
[somewhat later]
Victory!!! ;D :p - this command worked:
rity ~ # su - zabbix -c 'AUTOSSH_DEBUG=1 autossh -M0 -q -f -N -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 zabbix@xxx.xxx.xx.xx'
Also note: you many need to 'ps aux | grep autossh' and kill off any unneeded autossh sessions that were started during testing - these were likely causing me an issue.
Ok, so next is to have a start up script and load this up at boot. The second article, in the Gentoo Wiki, has info on this.
[MUCH LATER]
Argh this startup script option doesn't work right.
They suggest using the /etc/conf.d/local.start file - however this isn't right, because AutoSSH needs to be up before Zabbix or this connection seems to fail, and can't be restarted properly without restarting. Also, /etc/init.d/local.start is also started on nonetwork - check using :
# rc-update -s
(NOTE: this is Gentoo specific, and show the startup state)
It seems that Zabbix would not connect properly:
# netstat -anp | grep zabbix
tcp 0 0 127.0.0.1:10050 0.0.0.0:* LISTEN 25905/zabbix_agentd
tcp 0 0 127.0.0.1:20050 0.0.0.0:* LISTEN 26146/sshd: zabbix
tcp 0 0 ::1:20050 :::* LISTEN 26146/sshd: zabbix
tcp 0 0 ::ffff:64.###.##.###:22 ::ffff:24.###.##.:53822 ESTABLISHED 26144/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 15271790 26146/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 15271788 26144/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 15271787 26146/sshd: zabbixand then after stopping both the zabbix-server on the server, and the zabbix-agentd on the client, and then starting up the AutoSSH tunnel, and then starting the Zabbix Agentd, and then the Zabbix server process on the server, it works. :roll:
(on client) # /etc/init.d/zabbix-agentd stop
(on server)# /etc/init.d/zabbix-server stop
# su -l zabbix -c 'autossh -M0 -q -f -N -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 zabbix@64.251.22.129'
# /etc/init.d/zabbix-server start
(on client) # /etc/init.d/zabbix-agentd startIt looked like this after that:
# netstat -anp | grep zabbix
tcp 0 0 127.0.0.1:10050 0.0.0.0:* LISTEN 26484/zabbix_agentd
tcp 0 0 127.0.0.1:10050 127.0.0.1:56269 ESTABLISHED 26485/zabbix_agentd
tcp 0 0 127.0.0.1:56269 127.0.0.1:10050 ESTABLISHED 26430/sshd: zabbix
tcp 0 48 ::ffff:64.251.22.129:22 ::ffff:24.84.203.:59368 ESTABLISHED 26428/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 15273533 26430/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 15273531 26428/sshd: zabbix
unix 3 [ ] STREAM CONNECTED 15273530 26430/sshd: zabbixAlmost final victory - at least I know why its failing (I think) - the AutoSSH tunnel needs to be set up before the Zabbix server starts.
So now I need a decent startup script for AutoSSH, Gentoo style, which specifies that it should load before Zabbix-server.
In the Gentoo Wiki there a good article on this:
http://gentoo-wiki.com/HOWTO_Make_an_rc_script
So far I've come up with this - its not right, but at least for now ...
# cat /etc/init.d/autossh
#!/sbin/runscript
depend() {
# the daemon needs the internet to function
need net
# the daemon should run before zabbix-server
before zabbix-server
# the daemon uses logger
use logger
# the daemon should run after domainname
after domainname
# may write log file to disk (to be implemented)
need localmount
# needs to use pidfile
after bootmisc
}
checkconfig() {
if [ -z "$LOCAL_USER" ] || [ -z "$SSHKEY_FILE" ] || [ -z "$LOCALPORT_LOCALIP_CLIENTPORT" ] || [ -z "$CLIENT_USER_IP" ]; then
eerror "You must set config options in /etc/conf.d/autossh first"
return 1
fi
}
start() {
# display to the user what you're doing
ebegin "Starting autossh daemon"
#run the checkconfig function
checkconfig || return 1
# Start the process as a daemon and record the pid number
start-stop-daemon --start --background --pidfile /var/run/autossh.pid --make-pidfile --chuid $LOCAL_USER --exec /usr/bin/autossh -- -M0 -q -f -N -i $SSHKEY_FILE -L $LOCALPORT_LOCALIP_CLIENTPORT $CLIENT_USER_IP
# output success or failure
eend $?
}
stop() {
# display a message to the user
ebegin "Stopping autossh daemon"
# stop the daemon using the pid recorded from the start() function
start-stop-daemon --stop --pidfile /var/run/autossh.pid --name autossh
# output success or failure
eend $?
}And then this for the config file (some values changed):
# cat /etc/conf.d/autossh
#/etc/conf.d/autossh - configuration variables
LOCAL_USER="zabbix"
SSHKEY_FILE="/zabbix/home/.ssh/id_dsa"
LOCALPORT_LOCALIP_CLIENTPORT="20050:localhost:10050"
CLIENT_USER_IP="zabbix@xxx.xxx.xxx.xxx"Useful Links:
http://www.harding.motd.ca/autossh/README
http://linux.die.net/man/1/autossh
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4
Feedback awaiting moderation
This post has 73 feedbacks awaiting moderation...
