« Programming using my LAN step one PhpMyAdminAutomating some handy MythTV scripts »

Zabbix + ssh tunnel with AutoSSH

2007/12/10

  01:43:59 am, by   , 1494 words  
Categories: Gentoo Linux, Zabbix Monitoring

Zabbix + ssh tunnel with AutoSSH

I've spent a fair amount of time setting up a Zabbix system monitoring system to monitor the functioning of the LAN computers - Gentoo and Windows. Other than Zabbix 1.1.7 being a little tedious to set up, including the various add-on scanners I've configured, its fairly easy.

However, monitoring computers outside the LAN has been tricky, as I want to do it securely. Zabbix sends its info (at least in this version) through clear text, and I'd like to have in encrypted.

After several attempts at configuring AutoSSH (over several months :roll: ) I made some progress - first this article had some handy steps for testing:

http://www.linuxquestions.org/linux/answers/Networking/Forward_ports_via_an_SSH_tunnel_with_auto_reconnect_using_key_pair_authentication

namely:

Time to forward a port

a)add the port forward feature to the ssh command: '-L <local port>:system:<remote port>'
(b)'-L 1234:localhost:6667' would be my local system listening on port 1234 and sending requests to port 6667 of the remote's local loopback address (127.0.0.1)
(c)if forwarding is enabled on the remote system the you could use the remote system as a stepping stone to another.
(d)the new command would be:
(e)ssh -i <user's home dir>/.ssh/id_rsa -L <local port>:system:<remote port> <remote user>@<ip address>
(f)ie: 'ssh -i /root/.ssh/id_rsa -L 1234:localhost:6667 zxcv@207.46.19.30
(g)verify it works
(g)1.connect to the remote system
(g)2.set your irc client to connect locally on port 1234 which should forward the request to the remote system.

So, following these steps (I had already set up the RSA keys) I ended up with this command:

ssh -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 xxx.xxx.xx.xx

Finally, the Zabbix server process was getting results from the server outside the LAN:

Zabbix on CoLo Server
Zabbix on CoLo Server

With this config having been added to the '/etc/zabbix/zabbix_agentd.conf' file:

# List of comma delimited IP addresses (or hostnames) of ZABBIX servers.
# No spaces allowed. First entry is used for sending active checks.
# Note that hostnames must resolve hostname-[gt]IP address and
# IP address-[gt]hostname.
Server=127.0.0.1

# Server port for sending active checks
ServerPort=20050

# Unique hostname. Required for active checks.
Hostname=co-loserver

# Listen port. Default is 10050
ListenPort=10050

# IP address to bind agent
# If missing, bind to all available IPs
ListenIP=127.0.0.1

# Disable active checks. The agent will work in passive mode listening server.
DisableActive=1

I disabled active checks, mostly as a first stage, but I also suspected there might be issues trying to get the active checks back thorugh the firewall for the LAN.

The next step in that article, about setting up AutoSSH, didn't work for me. The logs were showing results like:

rity ~ # tail -n2000 /var/log/messages | grep autossh

Dec  9 22:33:06 rity autossh[27800]: port set to 0, monitoring disabled
Dec  9 22:33:06 rity autossh[27801]: checking for grace period, tries = 0
Dec  9 22:33:06 rity autossh[27801]: starting ssh (count 1)
Dec  9 22:33:06 rity autossh[27802]: execing /usr/bin/ssh
Dec  9 22:33:06 rity autossh[27801]: ssh child pid is 27802
Dec  9 22:33:06 rity autossh[27801]: check on child 27802
Dec  9 22:33:06 rity autossh[27801]: set alarm for 600 secs
Dec  9 22:33:08 rity autossh[27801]: check on child 27802
Dec  9 22:33:08 rity autossh[27801]: ssh exited prematurely with status 0; autossh exiting
Dec  9 22:33:08 rity autossh[27801]: expired child, returning 2

Eventually I found this entry (actually I had seen it before several times, but there was a new entry which made it all work):

http://gentoo-wiki.com/HOWTO_autossh

So, the command ends up looking like:

AUTOSSH_DEBUG=1 autossh -M0 -q -f -N -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 zabbix@xxx.xxx.xx.xx

I used the debug environment variable to assist in debugging any issues. I couldn't see any, finally (I've skipped over several false starts).

Here's what the log showed when AutoSSH was working properly:

Dec  9 23:45:22 rity autossh[16217]: port set to 0, monitoring disabled
Dec  9 23:45:22 rity autossh[16218]: checking for grace period, tries = 0
Dec  9 23:45:22 rity autossh[16218]: starting ssh (count 1)
Dec  9 23:45:22 rity autossh[16218]: ssh child pid is 16219
Dec  9 23:45:22 rity autossh[16218]: check on child 16219
Dec  9 23:45:22 rity autossh[16218]: set alarm for 600 secs
Dec  9 23:45:22 rity autossh[16219]: execing /usr/bin/ssh
Dec  9 23:55:22 rity autossh[16218]: check on child 16219
Dec  9 23:55:22 rity autossh[16218]: set alarm for 600 secs

I used this to look at what ports Zabbix was using before the ssh tunnel was set up:

SP29 ~ # netstat -anp | grep zabbix
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:10050         0.0.0.0:*               LISTEN      3999/zabbix_agentd

Here's what it looked like after the AutoSSH tunnel was going:

SP29 ~ # netstat -anp | grep zabbix
tcp        0      0 127.0.0.1:10050         0.0.0.0:*               LISTEN      8242/zabbix_agentd
tcp        0      0 64.251.22.129:22        24.84.203.91:3109       ESTABLISHED 14684/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     8748459 14686/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     8748457 14684/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     8748456 14686/sshd: zabbix

Still however the Zabbix server was NOT receiving data over the AutoSSH tunnel. Very disappointing. Still I had managed to get an SSH tunnel to work, from the command line. There must be a way.

[somewhat later]

Victory!!! ;D :p - this command worked:

rity ~ # su - zabbix -c 'AUTOSSH_DEBUG=1 autossh -M0 -q -f -N -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 zabbix@xxx.xxx.xx.xx'

Also note: you many need to 'ps aux | grep autossh' and kill off any unneeded autossh sessions that were started during testing - these were likely causing me an issue.

Ok, so next is to have a start up script and load this up at boot. The second article, in the Gentoo Wiki, has info on this.

[MUCH LATER]

Argh this startup script option doesn't work right.

They suggest using the /etc/conf.d/local.start file - however this isn't right, because AutoSSH needs to be up before Zabbix or this connection seems to fail, and can't be restarted properly without restarting. Also, /etc/init.d/local.start is also started on nonetwork - check using :

# rc-update -s

(NOTE: this is Gentoo specific, and show the startup state)

It seems that Zabbix would not connect properly:

 # netstat -anp | grep zabbix
tcp        0      0 127.0.0.1:10050         0.0.0.0:*               LISTEN      25905/zabbix_agentd
tcp        0      0 127.0.0.1:20050         0.0.0.0:*               LISTEN      26146/sshd: zabbix
tcp        0      0 ::1:20050               :::*                    LISTEN      26146/sshd: zabbix
tcp        0      0 ::ffff:64.###.##.###:22 ::ffff:24.###.##.:53822 ESTABLISHED 26144/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     15271790 26146/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     15271788 26144/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     15271787 26146/sshd: zabbix

and then after stopping both the zabbix-server on the server, and the zabbix-agentd on the client, and then starting up the AutoSSH tunnel, and then starting the Zabbix Agentd, and then the Zabbix server process on the server, it works. :roll:

(on client) # /etc/init.d/zabbix-agentd stop

(on server)# /etc/init.d/zabbix-server stop
# su -l zabbix -c 'autossh -M0 -q -f -N -i /zabbix/home/.ssh/id_dsa -L 20050:localhost:10050 zabbix@64.251.22.129'
# /etc/init.d/zabbix-server start

(on client) # /etc/init.d/zabbix-agentd start

It looked like this after that:

 # netstat -anp | grep zabbix
tcp        0      0 127.0.0.1:10050         0.0.0.0:*               LISTEN      26484/zabbix_agentd
tcp        0      0 127.0.0.1:10050         127.0.0.1:56269         ESTABLISHED 26485/zabbix_agentd
tcp        0      0 127.0.0.1:56269         127.0.0.1:10050         ESTABLISHED 26430/sshd: zabbix
tcp        0     48 ::ffff:64.251.22.129:22 ::ffff:24.84.203.:59368 ESTABLISHED 26428/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     15273533 26430/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     15273531 26428/sshd: zabbix
unix  3      [ ]         STREAM     CONNECTED     15273530 26430/sshd: zabbix

Almost final victory - at least I know why its failing (I think) - the AutoSSH tunnel needs to be set up before the Zabbix server starts.

So now I need a decent startup script for AutoSSH, Gentoo style, which specifies that it should load before Zabbix-server.

In the Gentoo Wiki there a good article on this:

http://gentoo-wiki.com/HOWTO_Make_an_rc_script

So far I've come up with this - its not right, but at least for now ...

 # cat /etc/init.d/autossh
#!/sbin/runscript

depend() {
        # the daemon needs the internet to function
        need net
        # the daemon should run before zabbix-server
        before zabbix-server
        # the daemon uses logger
        use logger
        # the daemon should run after domainname
        after domainname
  # may write log file to disk (to be implemented)
  need localmount
 # needs to use pidfile
  after bootmisc

}

checkconfig() {
        if [ -z "$LOCAL_USER" ] || [ -z "$SSHKEY_FILE" ] || [ -z "$LOCALPORT_LOCALIP_CLIENTPORT" ] || [ -z "$CLIENT_USER_IP" ]; then
          eerror "You must set config options in /etc/conf.d/autossh first"
          return 1
        fi
}

start() {
        # display to the user what you're doing
        ebegin "Starting autossh daemon"
        #run the checkconfig function
        checkconfig || return 1
        # Start the process as a daemon and record the pid number
        start-stop-daemon --start --background --pidfile /var/run/autossh.pid --make-pidfile --chuid $LOCAL_USER --exec /usr/bin/autossh -- -M0 -q -f -N -i $SSHKEY_FILE -L $LOCALPORT_LOCALIP_CLIENTPORT $CLIENT_USER_IP
        # output success or failure
        eend $?
}

stop() {
        # display a message to the user
        ebegin "Stopping autossh daemon"
        # stop the daemon using the pid recorded from the start() function
        start-stop-daemon --stop --pidfile /var/run/autossh.pid --name autossh
        # output success or failure
        eend $?
}

And then this for the config file (some values changed):

 # cat /etc/conf.d/autossh
#/etc/conf.d/autossh - configuration variables
LOCAL_USER="zabbix"
SSHKEY_FILE="/zabbix/home/.ssh/id_dsa"
LOCALPORT_LOCALIP_CLIENTPORT="20050:localhost:10050"
CLIENT_USER_IP="zabbix@xxx.xxx.xxx.xxx"

Useful Links:

http://www.harding.motd.ca/autossh/README
http://linux.die.net/man/1/autossh
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4

Feedback awaiting moderation

This post has 73 feedbacks awaiting moderation...


Form is loading...

June 2024
Mon Tue Wed Thu Fri Sat Sun
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
 << <   > >>
LAN / Networks related items ...

Search

  XML Feeds

multi-blog platform