Procurve Code Upgrade

June 29, 2012 By 2 Comments
showcmd

I just recently when through a round of code upgrade (at least 3 firmware versions each) on 300+ Procurve switches and so I thought I’d share with you what I had gone through.

My buddy Dean & I were tasked to upgrade a number of switches.  What we did was breaking them down into three phases spanning 3 days for a total of 16 hours window.  Although we had HP Network Automation (HPNA) available at our disposal, but I chose to do this manually due to a number of factors including code version inconsistency, boot room, primary vs secondary software image… Plus I also wanted to see how far we can push Pinkie so it was a perfect scenario doing the upgrade manually.

First, let’s talk about how we can push the images to the switches then we’ll go through the issues encountered during the upgrade and how to tackle them.  So, there are a number of ways you can get a firmware onto a HP Procurve switch. Let’s walk through them real quick:

1. X-modem: Slow, unsuitable for production environment upgrade.  Handy to recover corrupted flash.

2. USB Drive: Fast, efficient way to transfer, only available on certain code version.  Not suiteable large scale upgrade.

3. TFTP: Fast, efficient, requires IP connectivity.  Suiteable for production code upgrade.

Of the available options, only TFTP file transfer was suitable to do what we wanted to accomplished.  There were a number of TFTP servers available but being that I wrote Pinkie, I wouldn’t want to use any other TFTP server but Pinkie itself.

So how did we manage to upgrade 3 or more code versions on 300+ switches in under 16 hours change window?  Preparation.  Preparation was the key.  Prior to the actual change window, we staged both primary and secondary firmware with the next code versions.  If you are not familiar with Procurve firmware, they have to be upgraded sequentially since the firmware ties in with the boot-rom (I’ll explain in another blog post).  When the change window came, we used our terminal client and SSH’ed into the switches, rebooted them while having Pinkie constantly pinging them.  As soon as they came backup, we logged in again booted to the other image using the boot-system flash [primary/secondary] command.  After they came up the second time around, we pushed the third and final firmware (for most of them) to the switches using copy tftp flash command and rebooted again.  All the commands were prepared in little code snippets and pasted into the terminal client so it went rather smoothly.

At some point during the operation we had pushed Pinkie so hard (there were some 30-40 simultaneous TFTP requests going at the same time) that it locked up the user interface.  We thought for sure that it’s gonna crash but Pinkie hung in there and finished all TFTP file transfers even though the screen didn’t get updated for quite some time.

During the process, we ran into a number of issues.  One Procurve 3500 yl switch had corrupted flash.  We didn’t have time to repair the corrupted flash on the 3500 on the spot so we replaced the whole switch and opted to repair it later when we have some time on hand.  And it was successfully repaired using x-modem file transfer the next day.

There were a handful of 5406 yl switches that had corrupted flash also.  But for those, we were able to pull out the management cards and swapped out the flash memory then put them back to service.  The thing is you’ve gotta have a spare management module with the flash card that has firmware compatible with the boot rom on the bad management module.

Below are some commands that we used during the code upgrade process:

boot – reboot the switch to the current image.

show flash – show the firmware version on both primary and secondary boot images.

show version – show the running firmware and next boot image.

copy tftp flash… – initiates a file transfer request and copy the firmware from tftp server to the switch.

copy flash flash [primary/secondary] – copy the firmware from one slot to another.

That’s pretty much how we did it but like I said in the beginning, we broke it down into 3 phases.  The first phase we did it on a single POD of about 40 switches to get a feel for it then we double it on the second phase and finally on the third one, we did the rest of them.

One important thing I should note is that you need to look out for distribution switches – either do them first or do them last.  I overlooked one of them and as the result, I had to wait for it to finish before I could touch the aggregation & edge switches.  And it just happened that one of the distribution switches had corrupted flash so that delayed our code upgrade process longer than expected; although we did finish it under our reserved window for the change.

There you have it.  That’s how I did my firmware upgrade and the issue that I ran into during the process.  If you have any tips & tricks on how it could be done better, by all means, let me know.

Filed Under: TFTP Tagged With: , ,

Programmatically Ping a Networked Device

May 25, 2012 By 1 Comment

I would say Ping is one of the most popular methods used to troubleshoot network issue. When a networked device or server goes down, one of the first things a network engineer or a system admininistrator do is try to ping it and see if it can be reached.

This blog will show you two ways to ping a device so you can integrate the ping feature into your own application.

The first and easiest way to issue a ping command is to call the Ping method in the My.Computer.Network class:

If my.Computer.Network.Ping("www.google.com") Then
MsgBox("device up")
Else
MsgBox("device down")
End If

The drawback of this method is that it only gives you a boolean as the result to indicate whether the device is up or down. It doesn’t provide any other details like RTT, TTL or any error code for that matter.

The second way is to call the Ping.Send() method in the System.Net.NetworkInformation namespace which returns a PingReply object through which you could get more detailed information about the ping reply:

Dim myPing As New System.Net.NetworkInformation.Ping
Dim PR As System.Net.NetworkInformation.PingReply
PR = myPing.Send("www.yahoo.com")
If PR.Status = IPStatus.Success Then
MsgBox("Reply from " & PR.Address.ToString & ": BYTES=" & PR.Buffer.Length & " TIME<" & PR.RoundtripTime & "ms TTL=" & PR.Options.Ttl) Else MsgBox(PR.Status.ToString) End If

There you have it. Two ways of pinging a networked device. If you just need to check and see if a device is up or down, the first method might do just fine; however, if you need to gather some other statistics then obviously, the second one is the better choice.

I recommend that you read up on the PingReply object to get more information from the ping reply. If you wanted to implement some sort of Ping functionality as in Pinkie's then that's the way to go.

Filed Under: Ping, Visual Basic Tagged With: , , , , ,

Using PortScanner Feature in Pinkie More Effectively!

May 19, 2012 By 1 Comment

Another must have tool for a network professional is the Port Scanner. It is used to probe a host to see if host is listening on the specified port (open) or not.

Here’s how you can use PortScanner in Pinkie:

  • Scanning Default Ports: To scan the default ports, enter an IP Address, check the Default Ports checkbox then click on the Start Port Scan button.
  • Stop Port Scan: To stop a Port Scan in progress, click on the Stop Port Scan button and in a few seconds it should abort the scan completely.
  • Scan Ad Hoc Ports: To scan a host on an ad hoc basis, enter an IP Address, check the These Ports checkbox, enter the port numbers (separated by “,” & “;” for each port number or by “-” for a range of ports) then click on the Start Port Scan button.
  • Change Default Ports: Default Ports setting can be changed in the Settings Menu. Press F5 to bring up the Application Settings dialog then click on PortScanner tab. Enter new ports as you see fit.

PortScanner can still use a couple of enhancements. If you need something more specific, send in your request now.

Filed Under: PortScanner, Tips & Tricks Tagged With: , ,

Using Traceroute Feature in Pinkie More Effectively!

May 18, 2012 By Leave a Comment

Traceroute is a feature often used in network troubleshooting. Pinkie, as a suite network troubleshooting tools, has this feature well integrated into its One Window, On App user interface and also enable you to execute the trace automatically thus eliminating the manual work and save time.

There are a number of fine enhancements added to Pinkie’s Traceroute feature that you can’t find in other tools:

  • Copy The Host Address: If you want to copy the host address on the Traceroute result, just simply click on the hop and either the IP address or the hostname (depends on your prefence in the Settings menu) will be copied automatically.
  • Manually Start/Stop a Traceroute: Right click on a host in the Workspace, select Traceroute and click on Start/Stop menu item to start/stop the traceroute.
  • Copy Last Hop Address: At times, a network administrator might need to log into the last hop router to troubleshoot network issues. To get the last hop router, just right click the host, select Traceroute then click on Copy Last Hop to Clipboard. Pinkie will figures out the last hop that responded to the traceroute and copy the IP address to the clipboard which can be then pasted into your favorite Telnet or SSH client.
  • Copy Traceroute Result to Clipboard: There will be time when a network administrator needs to send the traceroute result to a third party. This can be done easily in Pinkie’s Traceroute function by right click on the host, select Traceroute then click on Copy Result to Clipboard. The entire traceroute result for that particular host will be copied to the clipboard.
  • Save Traceroute Result to a File: Similar to copy result, sometime, you might just want to save the traceroute result to disk. This can be done by right click on the host, select Traceroute then click on Save to File. A Save As dialog will popup prompting you for the location where you want to save it.

Those are just some basic enhancements of the Traceroute feature in Pinkie. I’ll cover the rest in another Tips & Tricks blog.

With these enhancements, I hope you will not go back to the Command Prompt again to do another manual traceroute.

Filed Under: Tips & Tricks, Traceroute Tagged With: , , , , , , , , ,