Cisco ASA Packet Capture

The ASA platform has fantastic built-in packet capture capabilities which can come in very handy for troubleshooting issues. I will be demonstrating some of the capabilities using an ASA 5505 running version 9.0(1).

Performing a packet capture is done using the capture command from privileged exec mode.

asa5505(config)# capture asa_cap ?

exec mode commands/options:
  access-list      Capture packets that match access-list
  buffer           Configure size of capture buffer, default is 512 KB
  circular-buffer  Overwrite buffer from beginning when full, default is non-circular
  ethernet-type    Capture Ethernet packets of a particular type, default is IP
  headers-only     Capture only L2, L3 and L4 headers of packet without data in them
  interface        Capture packets on a specific interface
  match            Capture packets matching five-tuple
  packet-length    Configure maximum length to save from each packet, default is 1518 bytes
  real-time        Display captured packets in real-time. Warning: using this option with a slow console connection may
                   result in an excessive amount of non-displayed packets due to performance limitations.
  trace            Trace the captured packets
  type             Capture packets based on a particular type
  

A common scenario is to determine if you are seeing bidirectional traffic to a host on the internet. For this example, I'm going to validate that I am seeing IP traffic to 8.8.8.8, one of Google's public DNS servers.

! We will start with defining an access-list to match the traffic

access-list asa_cap extended permit ip any4 host 8.8.8.8
access-list asa_cap extended permit ip host 8.8.8.8 any4

! Next we define the packet capture

capture asa_cap interface inside access-list asa_cap

We can verify the config by running 'show capture'.

capture asa_cap type raw-data access-list asa_cap interface inside [Capturing - 9648 bytes]

Looking at the results, you should notice that it automatically added the type 'raw-data' and also lists the amount of captured data. The raw-data type will capture any IP traffic that is transmitted through the ASA.
Now we can look at the captured data using 'show capture asa_cap'.

asa5505# sh cap asa_cap

177 packets captured

   1: 20:14:01.763006       802.1Q vlan#1 P0 10.1.1.11.44927 > 8.8.4.4.53:  udp 32
   2: 20:14:01.799824       802.1Q vlan#1 P0 8.8.4.4.53 > 10.1.1.11.44927:  udp 112
   3: 20:14:04.794056       802.1Q vlan#1 P0 10.1.1.11.52235 > 8.8.4.4.53:  udp 32
   4: 20:14:04.831988       802.1Q vlan#1 P0 8.8.4.4.53 > 10.1.1.11.52235:  udp 112
   5: 20:14:07.820529       802.1Q vlan#1 P0 10.1.1.11.48624 > 8.8.4.4.53:  udp 32
   6: 20:14:07.866303       802.1Q vlan#1 P0 8.8.4.4.53 > 10.1.1.11.48624:  udp 112

Each captured packet shows a timestamp of the time the packet was received, the VLAN ID since this is a 5505, the source IP and port, the destination IP and port, the type of packet, and the packet size. For TCP traffic it will also list the TCP MSS size which can be extremely helpful for troubleshooting MTU issues.
You can also examine a specific packet using the 'packet-number ' keyword and view some more packet details using the 'details' keyword. The details will include the TCP window size, TCP flags such as SYN and ACK, TTL, and if the DF bit is set.
If I want to do a quick packet capture, I typically skip the access-list altogether. You can do it inline using a 5-tuple match. A 5-tuple match uses 5 values to match a packet for the capture. Those values are protocol, source IP address, source port, destination IP address, and destination port. I also use the real-time keyword so that it will buffer the captured packets as well as follow them on the screen.
This time will capture packets to one of Google's web servers. Note that if you specify the capture traffic inline you do not need to specify the return traffic, it will match that automatically.

asa5505# capture asa_cap2 interface inside real-time match ip any host 74.125.225.32

Warning: using this option with a slow console connection may
         result in an excessive amount of non-displayed packets
         due to performance limitations.

Use ctrl-c to terminate real-time capture


   1: 20:35:14.573273       802.1Q vlan#1 P0 10.1.1.10.60295 > 74.125.225.32.80: S 1157924479:1157924479(0) win 8192 
   2: 20:35:14.574768       802.1Q vlan#1 P0 10.1.1.10.60296 > 74.125.225.32.80: S 3232879419:3232879419(0) win 8192 
   3: 20:35:14.614241       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: S 3782939017:3782939017(0) ack 1157924480 win 42900 
   4: 20:35:14.617476       802.1Q vlan#1 P0 10.1.1.10.60295 > 74.125.225.32.80: . ack 3782939018 win 68
   5: 20:35:14.618284       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60296: S 4201067853:4201067853(0) ack 3232879420 win 42900 
   6: 20:35:14.620130       802.1Q vlan#1 P0 10.1.1.10.60296 > 74.125.225.32.80: . ack 4201067854 win 68
   7: 20:35:14.649044       802.1Q vlan#1 P0 10.1.1.10.60295 > 74.125.225.32.80: P 1157924480:1157924818(338) ack 3782939018 win 68
   8: 20:35:14.688974       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . ack 1157924818 win 666
   9: 20:35:14.747443       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782939018:3782940278(1260) ack 1157924818 win 666
  10: 20:35:14.748343       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782940278:3782941538(1260) ack 1157924818 win 666
  11: 20:35:14.750434       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782941538:3782942798(1260) ack 1157924818 win 666
  12: 20:35:14.750479       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782942798:3782944058(1260) ack 1157924818 win 666
  13: 20:35:14.750510       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782944058:3782945318(1260) ack 1157924818 win 666
  14: 20:35:14.750556       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782945318:3782946578(1260) ack 1157924818 win 666
  15: 20:35:14.753348       802.1Q vlan#1 P0 10.1.1.10.60295 > 74.125.225.32.80: . ack 3782941538 win 68
  16: 20:35:14.753394       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782946578:3782947838(1260) ack 1157924818 win 666
  17: 20:35:14.753440       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782947838:3782949098(1260) ack 1157924818 win 666
  18: 20:35:14.753485       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782949098:3782950358(1260) ack 1157924818 win 666
  19: 20:35:14.753516       802.1Q vlan#1 P0 74.125.225.32.80 > 10.1.1.10.60295: . 3782950358:3782951618(1260) ack 1157924818 win 666

You can also use the packet capture functionality to see any packet drops on the firewall. This can be helpful if you are not seeing bidirectional traffic with a raw packet capture even though you are expecting it. This is done using the 'type asp-drop'. You can also specify they type of drop, such as acl_drop or no_route. For our purposes, we will capture all drop types.

capture drop_cap type asp-drop all

   1: 20:43:54.793934       802.1Q vlan#1 P0 10.1.1.11.34191 > 239.255.255.250.1900:  udp 125
   2: 20:43:55.827960       802.1Q vlan#1 P0 10.1.1.11.34191 > 239.255.255.250.1900:  udp 125 Drop-reason: (acl-drop) Flow is denied by configured rule
   3: 20:43:56.830462       802.1Q vlan#1 P0 10.1.1.11.34191 > 239.255.255.250.1900:  udp 125 Drop-reason: (acl-drop) Flow is denied by configured rule

You can see that my workstation is sending out multicast discovery messages for UPnP and my the ASA does not forward multicast packets by default, so it drops them.
At this point we have created a packet capture to capture raw packets matched by an access-list or inline. We have also captured dropped packets. But what if you want to examine the packet captures in even more details than what the ASA can provide? Thankfully the ASA has a way of downloading the pcap file. You can accomplish this through the copy command.

asa5505# copy /pcap capture:asa_cap2 tftp

Source capture name [asa_cap2]?

Address or name of remote host []? 10.1.1.10

Destination filename [asa_cap2]? asa_cap2.pcap
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
671 packets copied in 6.110 secs (111 packets/sec)

If you exclude the /pcap flag, the capture will still be uploaded to your tftp server, but it will be uploaded in the same format in which it's displayed using the 'show capture ' command. You can see below it displays the capture in Wireshark the same as it would if you ran the packet capture locally on your PC.

Click to enlarge

Other Thoughts

The packet capture is a powerful and useful tool that can be used to easily troubleshoot a lot of network and firewall issues. What I've demonstrated above is an overview of some of the capabilities. I urge you to explore deeper into this tool and discover everything else you can do. You can use it to capture different ethertype packets, encrypted and decrypted IPSec traffic if it is terminated at the ASA, webvpn traffic, and more.

One of my favorite uses is to prove (or disprove) proper NAT configuration. This was especially helpful when I was learning the changes that were first introduced in the 8.3 firmware. I will create two captures. One will capture traffic on the inside interface (on ingress) to a particular destination and one will capture traffic on the outside interface (on egress). The outside capture should show the outside local IP address as the source (and destination for return traffic). You can compare the two captures using timestamps and TCP data, such as source port. This can be a quick way to validate that your NAT configuration is good, or show that it needs some more work.

2014 New Years Resolutions

It's that time of year again. New Years traditionally is a time to reflect upon the year and make resolutions for the upcoming year. Like so many others, I have decided to make a blog post to document and motivate my goals for 2014. So here goes.

Professional Goals

1. Attain a better work/life balance. As is the nature of networking and working for a VAR, I find myself working after hours quite frequently. This makes it challenging to keep a healthy lifestyle outside of the workplace. This is going to be a work in progress all year, but I hope to find a good balance.

2. Blog more. I have several ideas saved up for future blog posts that I would like to write. The ultimate goal is to make it easier for younger people to enter and succeed in the field. As a side effect, it will help me learn as I further my learning and will be a reference for me in the future.

3. Attain CCNP R&S certification. I have the CCNP switch exam out of the way. I am currently studying for the routing exam and I purchased a block of Cisco Learning Lab hours. I would like pass both routing and tshoot exams in 2014.

Personal Goals

1. I would like to become more physically fit. I'm not really out of shape, but I don't work proactively to keep in shape. As I've discovered, the better I take care of myself the better I feel. I recently ordered a FitBit Force, which tracks fitness habits and sleep patterns.

2. Spend more time with family and friends. My family all lives close by, but I don't see them as often as I would like, maybe once or twice a month. My brother has three kids that I love to death and would love to see more often.

3. Vacation more often. This goes along with having a healthy work/life balance.I really should utilize all my given vacation time and use it to travel around the country and/or the world. I love travelling, but I'm often constrained by time and money, but short trips can be relatively inexpensive and are great at dealing with work stress.

So those are my resolutions for the year. Feel free to offer suggestions or add your own goals in the comments.

Nexus vPC with Nortel SMLT

My previous blog post was about dual homing a Nortel switch or switch stack with a pair of Nexus 5k or 7k devices. This post will expand upon that configuration and connect our pair of Nexus switches with two stacks of Nortel 5020 switches running in an SMLT configuration.

Here's the topology:

As you can see, we have two stacks of three Nortel switches used for aggregation and server access switches. This could alternatively be a pair of Passport 8600 switches. There are three interfaces configured as the inter-switch trunk (IST). The IST is similar to the vPC peer link. Each switch on the top stack is connected to Nexus01 and each switch on the bottom stack are connected to Nexus02.

Here's the IST configuration for the Nortel switches:

Nortel-Stack01

vlan name 1000 "IST_VLAN"

interface vlan 1000
ip address 10.10.10.1 255.255.255.252 2

vlan ports 1/45,2/45,3/45 tagging tagAll 
vlan members 1000 1/45,2/45,3/45
vlan ports 1/45,2/45,3/45 pvid 1000
vlan configcontrol strict
no auto-pvid

spanning-tree port 1/45,2/45,3/45 learning disable
mlt spanning-tree 1 stp 1 learning disable

mlt 1 name "IST" enable member 1/45,2/45,3/45 

interface mlt 1
ist peer-ip 10.10.10.2
ist vlan 1000
ist enable

vlacp macaddress 18:0c:20:00:00:0f
vlacp port 1/45,2/45,3/45 timeout short
vlacp port 1/45,2/45,3/45 enable
vlacp enable

Nortel-Stack02

vlan create 1000 type port 1
vlan name 1000 "IST_VLAN"

interface vlan 1000
ip address 10.10.10.2 255.255.255.252 2

vlan ports 1/45,2/45,3/45 tagging tagAll 
vlan members 1000 1/45,2/45,3/45
vlan ports 1/45,2/45,3/45 pvid 1000
vlan configcontrol strict
no auto-pvid

spanning-tree port 1/45,2/45,3/45 learning disable
mlt spanning-tree 1 stp 1 learning disable

mlt 1 name "IST" enable member 1/45,2/45,3/45 

interface mlt 1
ist peer-ip 10.10.10.1
ist vlan 1000
ist enable

vlacp macaddress 18:0c:20:00:00:0f
vlacp port 1/45,2/45,3/45 timeout short
vlacp port 1/45,2/45,3/45 enable
vlacp enable

As you can see, we are using VLAN 1000 for the IST connection, disabling spanning-tree for the IST ports, creating MLT ID 1 and designating it as an IST. The IST VLAN is used for SMLT exchanges and keepalives.

Since MLT/SMLT connections don't use any negoitiation mechanism to form the trunk, we need something that can detect a failure at layer 2 to layer 3. This is where VLACP comes in. VLACP is a Nortel proprietary protocol similar to LACP and used in conjunction with MLT and SMLT connections. Like LACP, if an interface fails to see hello packets from it's neighbor interface, it will bring that interface to a down state to prevent packet loss.

At this point, our IST should be established between the two switches. We need to configure the uplinks to the Nexus switches and assign VLANs to the trunks. We will be utilizing LACP for this connection.

On Both Switches

lacp key 32 mlt-id 32 smlt-id 32
lacp port-mode advance
lacp smlt-sys-id 18:0c:20:00:00:0f

interface FastEthernet 1/48,2/48,3/48
 lacp key 32
 lacp timeout-time short
 lacp mode active
 lacp aggregation enable

vlan create 2,4,8 type port 1
vlan ports 1/48,2/48,3/48 tagging untagPvidOnly
vlan members 1 1/45,1/48,2/45,2/48,3/45,3/48
vlan members 2 1/45,1/48,2/45,2/48,3/45,3/48
vlan members 4 1/45,1/48,2/45,2/48,3/45,3/48
vlan members 8 1/45,1/48,2/45,2/48,3/45,3/48

As you can see, this configuration is similar to the LACP configuration from a single Nortel switch with one noticeable difference. By default, LACP creates a system id based on the built-in MAC address of a switch. Since we are connecting two switch stacks to the Nexus switches, we need to define a shared system id to use. As shown above, we start by associating the lacp key with an MLT and SMLT id. We then assign an SMLT system id. You can just use the built-in address from one of the switches.

We already covered the configuration for the Nexus port-channels in the previous blog post. The configuration is identical except that we are bundling more interfaces.

At this point, everything should be connected and working properly. You can verify the LACP system ID on the Nexus switches by issuing the "show lacp neighbors" command. You can also verify that the port channels are live and passed the vPC consistency check with a "show vpc".

Nexus vPC to Nortel Switches

I have a customer that currently has a network with a pair of Nortel Passport 8000 switches as their core and a bunch of Nortel 5510 and 5520 access switches for server and workstation connectivity. This customer decided to purchase a pair of Nexus 5596T switches to replace the Nortel 8000 switches.

In the existing environment, they are utilizing Nortel's split multi-link trunking (SMLT) for layer 2 resiliency and aggregation. I have been tasked with setup of the new Nexus switches and configuring a similar topology using Cisco vPC technology for layer 2 resiliency and aggregation.

In this example, we are going to create vPC peer-link and a single LACP port channel to a Nortel 5510 switch.

Let's start with the basic Nexus vPC configuration. I'm not going to spend a lot of time on it as it's been covered by many people.

Nexus-01
feature vpc
feature lacp

interface mgmt0
  ip address 192.168.200.253/24

vpc domain 1
  role priority 4096
  peer-keepalive destination 192.168.200.253
  peer-gateway
  auto-recovery

interface Ethernet1/31
  descriptoin vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface Ethernet1/32
  description vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface vlan2
  ip address 192.168.10.254/24
  standby 2 ip 192.168.10.1
  standby 2 priority 110
  no shutdown

Nexus-02
feature vpc
feature lacp 

interface mgmt0
  ip address 192.168.200.254/24

vpc domain 1
  role priority 8192
  peer-keepalive destination 192.168.200.254
  peer-gateway
  auto-recovery

interface Ethernet1/31
  descriptoin vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface Ethernet1/32
  description vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface vlan2
  ip address 192.168.10.253/24
  standby 2 ip 192.168.10.1
  no shutdown

And now we confirm that the vPC peer-link is up and working properly.

Nexus-01# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 1
Peer Gateway                      : Enabled
Peer gateway excluded VLANs     : -
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     1-2

Nexus-02# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : secondary
Number of vPCs configured         : 1
Peer Gateway                      : Enabled
Peer gateway excluded VLANs     : -
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     1-2

And now that the vPC connection is up between the switches, we can configure the port-channel for the Nortel switch.

Nexus-01

interface Ethernet1/1
  description Trunk to Nortel Test
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description Trunk to Nortel Test
  no lacp graceful-convergence
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  vpc 10

Nexus-02

interface Ethernet1/1
  description Trunk to Nortel Test
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description Trunk to Nortel Test
  no lacp graceful-convergence
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  vpc 10

*Update: The configuration "no lacp graceful-convergence" was added to the port channel interfaces. LACP graceful convergence was introduced in version 5.1(3)N1(1) and is on by default. Cisco recommends disabling this feature on port channels connected to non NX-OS devices as it can cause the interfaces to inexplicibly enter a standby state.

And now for the multi-link trunk (MLT) configuration on the Nortel switch.

ip address switch 192.168.10.10
ip address netmask 255.255.255.0
vlan mgmt 2

vlan create 2 type port 1
vlan ports 47-48 tagging unTagPvidOnly
vlan members 1 1-48
vlan members 2 47-48
vlan ports 47-48 pvid 1

interface fastEthernet 47-48
lacp key 10
lacp timeout-time short
lacp mode active
lacp aggregation enable

Note that I left the PVID untagged for the two uplink ports to ensure compatibility with the Nexus switches since Cisco uses the concept of native VLANs on trunk ports. For this example, we are keeping it simple and using VLAN 1 for the native VLAN.

When you configure an LACP key on the Nortel switch, it assigns the unique LACP key to an MLT starting with the highest number MLT (32 in this case) and working backwards. The LACP key is local to the switch or stack and is similar to the channel-group with the Nexus switches. The MLT ID is similar to the port-channel interface on the Nexus switches.

At this point, the port-channel and MLT should be up on all the switches. Let's confirm.

Nexus-01# sh lacp neigh
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
PO10 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/1      32768,0-19-69-a6-4c-0  0x2f            378         SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x300a                      0x3f

Nexus-02# sh lacp neigh
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
Po10 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/1      32768,0-19-69-a6-4c-0  0x30            692         SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x300a                      0x3f

Nexus-01# sh vpc
!------Excluded for brevity---------!
vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     success                    1-2

Nexus-02# sh vpc
!------Excluded for brevity---------!
vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     success                    1-2

5510-48T-PWR# sh lacp port 47-48
                                  Admin Oper         Trunk Partner
Port Priority Lacp    A/I Timeout Key   Key   AggId Id    Port    Status
---- -------- ------- --- ------- ----- ----- ----- ----- ------- -------
47   32768    Active  A   Short   10    12298 8224   32    257     Active
48   32768    Active  A   Short   10    12298 8224   32    16641   Active

As you can see, LACP is up on all switches and vPC 10 is up on both Nexus switches. At this point we can confirm connectivity.

Nexus-01# ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10): 56 data bytes
64 bytes from 192.168.10.10: icmp_seq=0 ttl=63 time=1.697 ms
64 bytes from 192.168.10.10: icmp_seq=1 ttl=63 time=1.305 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=63 time=1.287 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=63 time=1.329 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=63 time=1.393 ms

Nexus-02# ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10): 56 data bytes
64 bytes from 192.168.10.10: icmp_seq=0 ttl=62 time=9.314 ms
64 bytes from 192.168.10.10: icmp_seq=1 ttl=62 time=3.379 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=62 time=3.349 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=62 time=3.354 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=62 time=3.352 ms

5510-48T-PWR>ping 192.168.10.1
Host is reachable
5510-48T-PWR>ping 192.168.10.254
Host is reachable
5510-48T-PWR>ping 192.168.10.253
Host is reachable

As we can see, ping works from each of the Nexus switches and from the Nortel switch to each Nexus switch and the shared HSRP IP.

The biggest caveat with this configuration is that you cannot change VLAN assignments on the trunk while LACP is active on the Nortel switch. If you change the allowed VLANs on either side of the trunk, it brings down the LACP neighborship until you disable and re-enable LACP on both uplinks. For this reason, I recommend you set the allowed VLANs on the port-channel on the Nexus switches to prevent the trunk from going down simply by creating a VLAN on the switch.

Forcing a Cisco ASA Reboot

I manage a good number of  Cisco ASA 5505 firewalls. I am currently working on one for a remote employee for a company who's network I manage. The firewall, at the time, was running 9.1(1)4. I discovered an issue with that code where the VPN process will occasionally cause the firewall to crash and become unresponsive for about 10-15 minutes. In an effort to update this firewall, which happens to be in Pennsylvania (I'm in Iowa), I uploaded the new firewall and attempted to reload the device. I type reload, hit enter twice to confirm, and nothing happens! I tried it several times with different flags, including reload quick and reload noconfirm. Nothing worked! I got so desperate I even opened ASDM and attempted to reload there, thinking it might have different hooks into the underlying OS. That failed as well. Desperate to get this firewall rebooted to fix the VPN bug (and this newly discovered reload bug, likely caused by the VPN bug), I came across the crashinfo command. To preface this, if you aren't familiar with ASAs, if they crash they dump a bunch of information to a text file on the flash drive called crash.txt. It contains a bunch of debug information, including the current memory contents and the process(es) that crashed. The crashinfo command allows you to view or save the crash info and also allows you to simulate a crash, either by doing a test or a forced crash as you can see from the output below.

asa(config)# crashinfo ?
configure mode commands/options:

console  Control output of crashinfo to the console
save     Save

exec mode commands/options:
force       Forcibly crash the system and reboot
test        Test crashinfo generation - will not crash the system

asa(config)# crashinfo force ?exec mode commands/options:
page-fault    Crash by causing a page fault exception
watchdog      Crash by causing a watchdog timeout

So to force a reload on the firewall, I was able to issue:

crashinfo force watchdog

and the firewall immediately rebooted and came up on the new firmware. I wouldn't recommend doing this unless you absolutely have to, but it did bail me out in this situation.

Introduction

My name is Mike. I am currently a technical consultant/network engineer for an MSP and Service Provider in Central Iowa. I have been working in this industry for about 3 years and I am currently the lead network engineer working primarily with Cisco routers, switches, and firewalls. Due to the nature of the business however, I do find myself working with a vast range of vendors, including HP, Juniper, Dell (formerly Force10 and PowerConnect), Fortinet, Foundry, Nortel, and so on. I also work quite a bit with storage and virtualization.

The purpose of this blog is two-fold. The first is to keep a personal repository of different issues I have come upon and new technologies I learn and implement. The second purpose is the offer yet another location where fellow networkers coming up in the business can hopefully benefit from some of the topics I post about. I realize there are already more blogs than I can count, but I'm hoping to offer my own perspective and I will hopefully be beneficial to some people.

If anyone has any questions about any blog posts or anything else, feel free to reach out to be at blog@networknerdmike.com or on Twitter @networknerdmike.