Sunday, December 29, 2013

2014 New Years Resolutions

It's that time of year again. New Years traditionally is a time to reflect upon the year and make resolutions for the upcoming year. Like so many others, I have decided to make a blog post to document and motivate my goals for 2014. So here goes.

Professional Goals

1. Attain a better work/life balance. As is the nature of networking and working for a VAR, I find myself working after hours quite frequently. This makes it challenging to keep a healthy lifestyle outside of the workplace. This is going to be a work in progress all year, but I hope to find a good balance.

2. Blog more. I have several ideas saved up for future blog posts that I would like to write. The ultimate goal is to make it easier for younger people to enter and succeed in the field. As a side effect, it will help me learn as I further my learning and will be a reference for me in the future.

3. Attain CCNP R&S certification. I have the CCNP switch exam out of the way. I am currently studying for the routing exam and I purchased a block of Cisco Learning Lab hours. I would like pass both routing and tshoot exams in 2014.

Personal Goals

1. I would like to become more physically fit. I'm not really out of shape, but I don't work proactively to keep in shape. As I've discovered, the better I take care of myself the better I feel. I recently ordered a FitBit Force, which tracks fitness habits and sleep patterns.

2. Spend more time with family and friends. My family all lives close by, but I don't see them as often as I would like, maybe once or twice a month. My brother has three kids that I love to death and would love to see more often.

3. Vacation more often. This goes along with having a healthy work/life balance.I really should utilize all my given vacation time and use it to travel around the country and/or the world. I love travelling, but I'm often constrained by time and money, but short trips can be relatively inexpensive and are great at dealing with work stress.

So those are my resolutions for the year. Feel free to offer suggestions or add your own goals in the comments.

Tuesday, October 1, 2013

Nexus vPC with Nortel SMLT

My previous blog post was about dual homing a Nortel switch or switch stack with a pair of Nexus 5k or 7k devices. This post will expand upon that configuration and connect our pair of Nexus switches with two stacks of Nortel 5020 switches running in an SMLT configuration.

Here's the topology:


As you can see, we have two stacks of three Nortel switches used for aggregation and server access switches. This could alternatively be a pair of Passport 8600 switches. There are three interfaces configured as the inter-switch trunk (IST). The IST is similar to the vPC peer link. Each switch on the top stack is connected to Nexus01 and each switch on the bottom stack are connected to Nexus02.

Here's the IST configuration for the Nortel switches:

Nortel-Stack01

vlan name 1000 "IST_VLAN"

interface vlan 1000
ip address 10.10.10.1 255.255.255.252 2

vlan ports 1/45,2/45,3/45 tagging tagAll 
vlan members 1000 1/45,2/45,3/45
vlan ports 1/45,2/45,3/45 pvid 1000
vlan configcontrol strict
no auto-pvid

spanning-tree port 1/45,2/45,3/45 learning disable
mlt spanning-tree 1 stp 1 learning disable

mlt 1 name "IST" enable member 1/45,2/45,3/45 

interface mlt 1
ist peer-ip 10.10.10.2
ist vlan 1000
ist enable

vlacp macaddress 18:0c:20:00:00:0f
vlacp port 1/45,2/45,3/45 timeout short
vlacp port 1/45,2/45,3/45 enable
vlacp enable

Nortel-Stack02

vlan create 1000 type port 1
vlan name 1000 "IST_VLAN"

interface vlan 1000
ip address 10.10.10.2 255.255.255.252 2

vlan ports 1/45,2/45,3/45 tagging tagAll 
vlan members 1000 1/45,2/45,3/45
vlan ports 1/45,2/45,3/45 pvid 1000
vlan configcontrol strict
no auto-pvid

spanning-tree port 1/45,2/45,3/45 learning disable
mlt spanning-tree 1 stp 1 learning disable

mlt 1 name "IST" enable member 1/45,2/45,3/45 

interface mlt 1
ist peer-ip 10.10.10.1
ist vlan 1000
ist enable

vlacp macaddress 18:0c:20:00:00:0f
vlacp port 1/45,2/45,3/45 timeout short
vlacp port 1/45,2/45,3/45 enable
vlacp enable

As you can see, we are using VLAN 1000 for the IST connection, disabling spanning-tree for the IST ports, creating MLT ID 1 and designating it as an IST. The IST VLAN is used for SMLT exchanges and keepalives.

Since MLT/SMLT connections don't use any negoitiation mechanism to form the trunk, we need something that can detect a failure at layer 2 to layer 3. This is where VLACP comes in. VLACP is a Nortel proprietary protocol similar to LACP and used in conjunction with MLT and SMLT connections. Like LACP, if an interface fails to see hello packets from it's neighbor interface, it will bring that interface to a down state to prevent packet loss.

At this point, our IST should be established between the two switches. We need to configure the uplinks to the Nexus switches and assign VLANs to the trunks. We will be utilizing LACP for this connection.

On Both Switches

lacp key 32 mlt-id 32 smlt-id 32
lacp port-mode advance
lacp smlt-sys-id 18:0c:20:00:00:0f

interface FastEthernet 1/48,2/48,3/48
 lacp key 32
 lacp timeout-time short
 lacp mode active
 lacp aggregation enable

vlan create 2,4,8 type port 1
vlan ports 1/48,2/48,3/48 tagging untagPvidOnly
vlan members 1 1/45,1/48,2/45,2/48,3/45,3/48
vlan members 2 1/45,1/48,2/45,2/48,3/45,3/48
vlan members 4 1/45,1/48,2/45,2/48,3/45,3/48
vlan members 8 1/45,1/48,2/45,2/48,3/45,3/48

As you can see, this configuration is similar to the LACP configuration from a single Nortel switch with one noticeable difference. By default, LACP creates a system id based on the built-in MAC address of a switch. Since we are connecting two switch stacks to the Nexus switches, we need to define a shared system id to use. As shown above, we start by associating the lacp key with an MLT and SMLT id. We then assign an SMLT system id. You can just use the built-in address from one of the switches.

We already covered the configuration for the Nexus port-channels in the previous blog post. The configuration is identical except that we are bundling more interfaces.

At this point, everything should be connected and working properly. You can verify the LACP system ID on the Nexus switches by issuing the "show lacp neighbors" command. You can also verify that the port channels are live and passed the vPC consistency check with a "show vpc".

Sunday, September 22, 2013

Nexus vPC to Nortel Switches

I have a customer that currently has a network with a pair of Nortel Passport 8000 switches as their core and a bunch of Nortel 5510 and 5520 access switches for server and workstation connectivity. This customer decided to purchase a pair of Nexus 5596T switches to replace the Nortel 8000 switches.

In the existing environment, they are utilizing Nortel's split multi-link trunking (SMLT) for layer 2 resiliency and aggregation. I have been tasked with setup of the new Nexus switches and configuring a similar topology using Cisco vPC technology for layer 2 resiliency and aggregation.

In this example, we are going to create vPC peer-link and a single LACP port channel to a Nortel 5510 switch.



Let's start with the basic Nexus vPC configuration. I'm not going to spend a lot of time on it as it's been covered by many people.

Nexus-01
feature vpc
feature lacp

interface mgmt0
  ip address 192.168.200.253/24

vpc domain 1
  role priority 4096
  peer-keepalive destination 192.168.200.253
  peer-gateway
  auto-recovery

interface Ethernet1/31
  descriptoin vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface Ethernet1/32
  description vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface vlan2
  ip address 192.168.10.254/24
  standby 2 ip 192.168.10.1
  standby 2 priority 110
  no shutdown

Nexus-02
feature vpc
feature lacp 

interface mgmt0
  ip address 192.168.200.254/24

vpc domain 1
  role priority 8192
  peer-keepalive destination 192.168.200.254
  peer-gateway
  auto-recovery

interface Ethernet1/31
  descriptoin vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface Ethernet1/32
  description vPC Peer Link
  switchport mode trunk
  spanning-tree port type network
  channel-group 1 mode active

interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface vlan2
  ip address 192.168.10.253/24
  standby 2 ip 192.168.10.1
  no shutdown

And now we confirm that the vPC peer-link is up and working properly.

Nexus-01# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 1
Peer Gateway                      : Enabled
Peer gateway excluded VLANs     : -
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     1-2

Nexus-02# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : secondary
Number of vPCs configured         : 1
Peer Gateway                      : Enabled
Peer gateway excluded VLANs     : -
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     1-2

And now that the vPC connection is up between the switches, we can configure the port-channel for the Nortel switch.
Nexus-01

interface Ethernet1/1
  description Trunk to Nortel Test
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description Trunk to Nortel Test
  no lacp graceful-convergence
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  vpc 10

Nexus-02

interface Ethernet1/1
  description Trunk to Nortel Test
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description Trunk to Nortel Test
  no lacp graceful-convergence
  switchport trunk allowed vlan 1-2
  switchport mode trunk
  vpc 10

*Update: The configuration "no lacp graceful-convergence" was added to the port channel interfaces. LACP graceful convergence was introduced in version 5.1(3)N1(1) and is on by default. Cisco recommends disabling this feature on port channels connected to non NX-OS devices as it can cause the interfaces to inexplicibly enter a standby state.

And now for the multi-link trunk (MLT) configuration on the Nortel switch.
ip address switch 192.168.10.10
ip address netmask 255.255.255.0
vlan mgmt 2

vlan create 2 type port 1
vlan ports 47-48 tagging unTagPvidOnly
vlan members 1 1-48
vlan members 2 47-48
vlan ports 47-48 pvid 1

interface fastEthernet 47-48
lacp key 10
lacp timeout-time short
lacp mode active
lacp aggregation enable

Note that I left the PVID untagged for the two uplink ports to ensure compatibility with the Nexus switches since Cisco uses the concept of native VLANs on trunk ports. For this example, we are keeping it simple and using VLAN 1 for the native VLAN.

When you configure an LACP key on the Nortel switch, it assigns the unique LACP key to an MLT starting with the highest number MLT (32 in this case) and working backwards. The LACP key is local to the switch or stack and is similar to the channel-group with the Nexus switches. The MLT ID is similar to the port-channel interface on the Nexus switches.

At this point, the port-channel and MLT should be up on all the switches. Let's confirm.

Nexus-01# sh lacp neigh
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
PO10 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/1      32768,0-19-69-a6-4c-0  0x2f            378         SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x300a                      0x3f

Nexus-02# sh lacp neigh
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
Po10 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/1      32768,0-19-69-a6-4c-0  0x30            692         SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x300a                      0x3f

Nexus-01# sh vpc
!------Excluded for brevity---------!
vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     success                    1-2

Nexus-02# sh vpc
!------Excluded for brevity---------!
vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     success                    1-2

5510-48T-PWR# sh lacp port 47-48
                                  Admin Oper         Trunk Partner
Port Priority Lacp    A/I Timeout Key   Key   AggId Id    Port    Status
---- -------- ------- --- ------- ----- ----- ----- ----- ------- -------
47   32768    Active  A   Short   10    12298 8224   32    257     Active
48   32768    Active  A   Short   10    12298 8224   32    16641   Active

As you can see, LACP is up on all switches and vPC 10 is up on both Nexus switches. At this point we can confirm connectivity.
Nexus-01# ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10): 56 data bytes
64 bytes from 192.168.10.10: icmp_seq=0 ttl=63 time=1.697 ms
64 bytes from 192.168.10.10: icmp_seq=1 ttl=63 time=1.305 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=63 time=1.287 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=63 time=1.329 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=63 time=1.393 ms

Nexus-02# ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10): 56 data bytes
64 bytes from 192.168.10.10: icmp_seq=0 ttl=62 time=9.314 ms
64 bytes from 192.168.10.10: icmp_seq=1 ttl=62 time=3.379 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=62 time=3.349 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=62 time=3.354 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=62 time=3.352 ms

5510-48T-PWR>ping 192.168.10.1
Host is reachable
5510-48T-PWR>ping 192.168.10.254
Host is reachable
5510-48T-PWR>ping 192.168.10.253
Host is reachable

As we can see, ping works from each of the Nexus switches and from the Nortel switch to each Nexus switch and the shared HSRP IP.

The biggest caveat with this configuration is that you cannot change VLAN assignments on the trunk while LACP is active on the Nortel switch. If you change the allowed VLANs on either side of the trunk, it brings down the LACP neighborship until you disable and re-enable LACP on both uplinks. For this reason, I recommend you set the allowed VLANs on the port-channel on the Nexus switches to prevent the trunk from going down simply by creating a VLAN on the switch.

Forcing a Cisco ASA Reboot

I manage a good number of  Cisco ASA 5505 firewalls. I am currently working on one for a remote employee for a company who's network I manage. The firewall, at the time, was running 9.1(1)4. I discovered an issue with that code where the VPN process will occasionally cause the firewall to crash and become unresponsive for about 10-15 minutes. In an effort to update this firewall, which happens to be in Pennsylvania (I'm in Iowa), I uploaded the new firewall and attempted to reload the device. I type reload, hit enter twice to confirm, and nothing happens! I tried it several times with different flags, including reload quick and reload noconfirm. Nothing worked! I got so desperate I even opened ASDM and attempted to reload there, thinking it might have different hooks into the underlying OS. That failed as well. Desperate to get this firewall rebooted to fix the VPN bug (and this newly discovered reload bug, likely caused by the VPN bug), I came across the crashinfo command. To preface this, if you aren't familiar with ASAs, if they crash they dump a bunch of information to a text file on the flash drive called crash.txt. It contains a bunch of debug information, including the current memory contents and the process(es) that crashed. The crashinfo command allows you to view or save the crash info and also allows you to simulate a crash, either by doing a test or a forced crash as you can see from the output below.

asa(config)# crashinfo ?
configure mode commands/options:

console  Control output of crashinfo to the console
save     Save

exec mode commands/options:
force       Forcibly crash the system and reboot
test        Test crashinfo generation - will not crash the system

asa(config)# crashinfo force ?exec mode commands/options:
page-fault    Crash by causing a page fault exception
watchdog      Crash by causing a watchdog timeout

So to force a reload on the firewall, I was able to issue:
crashinfo force watchdog

and the firewall immediately rebooted and came up on the new firmware. I wouldn't recommend doing this unless you absolutely have to, but it did bail me out in this situation.

Introduction

My name is Mike. I am currently a technical consultant/network engineer for an MSP and Service Provider in Central Iowa. I have been working in this industry for about 3 years and I am currently the lead network engineer working primarily with Cisco routers, switches, and firewalls. Due to the nature of the business however, I do find myself working with a vast range of vendors, including HP, Juniper, Dell (formerly Force10 and PowerConnect), Fortinet, Foundry, Nortel, and so on. I also work quite a bit with storage and virtualization.

The purpose of this blog is two-fold. The first is to keep a personal repository of different issues I have come upon and new technologies I learn and implement. The second purpose is the offer yet another location where fellow networkers coming up in the business can hopefully benefit from some of the topics I post about. I realize there are already more blogs than I can count, but I'm hoping to offer my own perspective and I will hopefully be beneficial to some people.

If anyone has any questions about any blog posts or anything else, feel free to reach out to be at blog@networknerdmike.com or on Twitter @networknerdmike.