*DAMN YOU ASA!!! * I recently set up a tac_plus daemon (running on an Ubuntu Server 12.10) to run AAA for our network. Everything works as it should except one switch is having issues authenticating when I set the tacacs source interface to loopback0 (with no source set it connects fine via one of the vlans). Loopback0 is very much reachable to/from the tacacs server. There is an ASA between these devices but it's not doing anything (basically allow any:any).

This is the config on the switch itself which works fine as is:

switch#sh run | i tacacs

aaa authentication login default group tacacs  local
aaa authentication login console group tacacs  local
aaa authentication enable default group tacacs  enable
aaa accounting exec default start-stop group tacacs 
aaa accounting commands 15 default start-stop group tacacs 
tacacs-server host 1.1.1.1
no tacacs-server directed-request
tacacs-server key 7 xxxxxxxx    

But once I add in:

switch(config)#ip tacacs source l0    

I can no longer log in with tacacs credentials (it fails over nicely to local though).

There are no ACLs set up in the tac_plus daemon for the user I am attempting to log in as or even any blanket ACLs. I reverted to the most basic of configs for tac_plus to test this.

I can SSH from the switch to the tacacs server. Interestingly enough, since ssh source is set to loopback0 I can see this from the tacacs server.

ohai@tacacs:~$ who
ohai pts/0        2013-01-18 07:18 (9.9.9.9)
ohai pts/1        2013-01-18 10:53 (2.2.2.2)    

Right there I am on pts/1 from the switch loopback0 IP.

So I started doing a little debugging on the switch.

switch(config)#do debug tacacs authent
TACACS  authentication debugging is on
switch(config)#ip tacacs source l0
switch(config)#
.Jan 18 21:08:03.338 GMT: TPLUS: Queuing AAA Authentication request 34 for processing
.Jan 18 21:08:03.338 GMT: TPLUS: processing authentication start request id 34
.Jan 18 21:08:03.338 GMT: TPLUS: Authentication start packet created for 34(ohai)
.Jan 18 21:08:03.338 GMT: TPLUS: Using server 1.1.1.1
.Jan 18 21:08:03.338 GMT: TPLUS(00000022)/0/NB_WAIT/57D5FF8: Started 5 sec timeout
.Jan 18 21:08:04.160 GMT: TPLUS(00000022)/0/NB_WAIT: socket event 2
.Jan 18 21:08:04.160 GMT: TPLUS(00000022)/0/NB_WAIT: wrote entire 51 bytes request
.Jan 18 21:08:04.160 GMT: TPLUS(00000022)/0/READ: socket event 1
.Jan 18 21:08:04.160 GMT: TPLUS(00000022)/0/READ: Would block while reading
.Jan 18 21:08:08.338 GMT: TPLUS(00000022)/0/READ/57D5FF8: timed out
.Jan 18 21:08:08.338 GMT: TPLUS: Authentication start packet created for 34(ohai)
.Jan 18 21:08:08.338 GMT: TPLUS(00000022)/0/READ/57D5FF8: timed out, clean up
.Jan 18 21:08:08.338 GMT: TPLUS(00000022)/0/57D5FF8: Processing the reply packet
.Jan 18 21:08:16.324 GMT: TPLUS: Queuing AAA Authentication request 34 for processing
.Jan 18 21:08:16.324 GMT: TPLUS: processing authentication start request id 34
.Jan 18 21:08:16.324 GMT: TPLUS: Authentication start packet created for 34(ohai)
.Jan 18 21:08:16.324 GMT: TPLUS: Using server 1.1.1.1
.Jan 18 21:08:16.332 GMT: TPLUS(00000022)/0/NB_WAIT/571C30C: Started 5 sec timeout
.Jan 18 21:08:17.003 GMT: TPLUS(00000022)/0/NB_WAIT: socket event 2
.Jan 18 21:08:17.003 GMT: TPLUS(00000022)/0/NB_WAIT: wrote entire 51 bytes request
.Jan 18 21:08:17.003 GMT: TPLUS(00000022)/0/READ: socket event 1
.Jan 18 21:08:17.003 GMT: TPLUS(00000022)/0/READ: Would block while readingdo u all
All possible debugging has been turned off
switch(config)#

I tried logging in while this was running and it seems as though it's trying but just not getting anything back. Nothing appears in the tac_plus log files (even with debugging set to 256, packet transmission/reception). Also interesting, netstat sees the connection.

root@tacacs:/var/log# netstat -tap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 localhost:mysql         *:*                     LISTEN      1019/mysqld
tcp        0      0 *:http                  *:*                     LISTEN      1163/apache2
tcp        0      0 *:tacacs                *:*                     LISTEN      3977/tac_plus
tcp        0      0 1.1.1.1:tacacs     2.2.2.2:23062     SYN_RECV    -
tcp        0      0 1.1.1.1:tacacs     2.2.2.2:36616     SYN_RECV    -
tcp        0      0 *:ssh                   *:*                     LISTEN      829/sshd
tcp        0      0 1.1.1.1:tacacs     3.3.3.3:31975     TIME_WAIT   -
tcp       52      0 1.1.1.1:ssh        9.9.9.9:4938       ESTABLISHED 3226/sshd: ohai.
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      829/sshd
root@tacacs:/var/log#

It would appear that the query is making it to the tacacs server but not to the tac_plus daemon itself but only from this one IP. I can set the source to be that default vlan that it connects from but that's pretty far from ideal (if it goes down for whatever reason) and I don't want unneeded exceptions to my SOP.

So I am officially at a loss. Anybody got any ideas?

all IPs, usernames, keys have been changed to protect the innocent! whoopsie, missed a couple :(