Hopefully if your reading this it’s not because your experiencing issue with one or more of your DSP farms. Unfortunately for me that’s exactly why I felt the need to write this post. The interesting thing was the problem didn’t actually manifest itself until I began sharing ISDN PSTN gateway resources with another remote location that was experiencing major issue with their POTS lines. This may or may not have had anything to do with the DSP I started having, but the coincidence was strange to say the least.
Host Port Interface
The Voice DSP control message logger feature provides improved debugging capabilities through IOS software and allows logging of control messages as they pass through the voice DSP firmware on the Host Port Interface or HPI. The logged messages can later be examined for diagnosis of voice problems.
There are two main types of HPI messages that flow through the HPI interface: control messages and data messages. Control messages carry control information between Cisco IOS software and the DSP. Data messages carry voice data.
The Voice DSP Contol Message Logger feature captures control messages sent between the platform-independent portions of Cisco IOS software and the DSP. The HPI subsystem that is in Cisco IOS software contains the platform-independent portion of Cisco IOS software. This feature addresses the sequence and contents of the control messages. The logged messages can be checked for parameters that might cause undesirable DSP behavior including the following.
Before we get started with configuration and monitoring using HPI, and DSP captures lets review setting up a DSP Farm within Call Manager and on the Router itself. If your not familiar with configuring Media Resources within Call Manager you can find them underneath the Menu Media Resources / Conference Bridge / Transcoder / Media Termination.
After you have configured the Transcoding, Media Termination, and Conference Bridging within Call Manager you can move onto the actual DSP Farm configuration on the Router itself. One of the interesting things about the DSP Farm is the fact that it registers itself much the same way a SCCP Phone would register with Call Manager. I’m not going to go into the actual configuration of the DSP Farm on the Router itself, but you can checkout my post on DSP and the related Farm configuration.
The RMA Replacement
When this all started I was convinced that the problem had to be a hardware related either with the PVDM’s themselves or the Router itself. I opened a TAC case with Cisco and I eventually convinced them to send replacement PVDM’s along with the Router chassis.
Now lets take a look at healthy DSP Farm with two 32 channel PVDM3 for a grand total of 64 channels.
R5#show voice dsp group all DSP groups on slot 0: dsp 1: State: UP, firmware: 35.1.8 Max signal/voice channel: 32/32 Max credits: 480, Voice credits: 480, Video credits: 0 num_of_sig_chnls_allocated: 29 Transcoding channels allocated: 1 Group: FLEX_GROUP_VOICE, complexity: FLEX Shared credits: 445, reserved credits: 0 Signaling channels allocated: 29 Voice channels allocated: 0 Credits used (rounded-up): 0 Group: FLEX_GROUP_XCODE, complexity: HIGH Shared credits: 0, reserved credits: 35 Transcoding channels allocated: 0 Credits used (rounded-up): 0 Slot: 0 Device idx: 0 PVDM Slot: 0 Dsp Type: SP2600 dsp 2: State: UP, firmware: 35.1.8 Max signal/voice channel: 32/32 Max credits: 480, Voice credits: 480, Video credits: 0 num_of_sig_chnls_allocated: 0 Transcoding channels allocated: 9 Group: FLEX_GROUP_VOICE, complexity: FLEX Shared credits: 75, reserved credits: 0 Signaling channels allocated: 0 Voice channels allocated: 0 Credits used (rounded-up): 0 Group: FLEX_GROUP_XCODE, complexity: HIGH Shared credits: 0, reserved credits: 309 Transcoding channels allocated: 0 Credits used (rounded-up): 0 Group: FLEX_GROUP_CONF, complexity: CONFERENCE Shared credits: 0, reserved credits: 96 Codec: CONF_G729, maximum participants: 8 Sessions per dsp: 5 Slot: 0 Device idx: 0 PVDM Slot: 0 Dsp Type: SP2600 DSP groups on slot 1: This command is not applicable to slot 1 0 DSP resource allocation failure
As you can see from the above output the status of the DSP Farm that is both 32 Channel PVDM-3 32 are up with a status of “UP, firmware: 35.1.8“. This is what a typical healthy DSP Farm should look like.
Now lets take a look at the state of the DSP Farm after a firmware crash
R5#show voice dsp group all | include State State: APP_DNLDING, firmware 0.0.0 State: APP_DNLDING, firmware 0.0.0 R5#
Preparing For The Crash
If you have a reoccurring issue with a DSP Farm crashing you can setup your Router to catch the crash when it happens the the flow the PVDM with the Host Port Interface capture. Personally I prefer to keep the logging off box so I’m going to use FTP to store the DSP Crash file when and if it occurs. So first things first lets configure and IP Host on the R5 that can be used to store the crash file. There are a couple of available options when configuring the Crash Dump. The destination can be pointed at local Flash, TFTP, FTP etc. The second option is the number of files to create during the crash.
- The destination DSP for the crash dump file or URL
- The file limit or number of crash dump files to write on server
Lets jump into the IP Host FTP Server configuration on R5 that we can use for the Crash Dump destination. The following is of course dependent on the FTP Server configuration.
R5#configure terminal R5(config)#ip ftp username dsp R5(config)#ip ftp password crash R5(config)#ip host server 172.16.21.20 R5(config)#
Now lets move forward with configuring the Crash Dump configuration on Router R5
R5(config)# R5(config)#voice dsp crash-dump file-limit 10 R5(config)#voice dsp crash-dump destination ftp://server/DSP-Crash.log R5(config)#end R5#
Now lets take a look at the Crash Dump configuration on R5
R5#show voice dsp crash-dump
Voice DSP Crash-dump status:
Destination file url is
ftp://server/DSP-Crash.log
File limit is 10
Next DSP dump file written will be
ftp://server/DSP-Crash1.log
Now lets move onto configuring the Host Port Interface or HPI capture. There are basically 3 options available when configuring the Host Port Interface capture.
- The HPI buffer used for the control message capture size from 872 – 60000000
- The destination HPI control message capture data destination or URL
- The periodic file duration that can be specified over time
Lets get started configuring the HPI with a destination pointed at the SERVER FTP URL
R5#configure terminal R5(config)#voice hpi capture buffer 1024 R5(config)#voice hpi capture destination ftp://server/HPI-CAP.log R5(config)#end R5#
Now that we have the HPI Capture configured lets take a look
R5#show voice hpi cap B65RTR#show voice hpi capture HPI Capture is on and is logging to URL ftp://server/HPI-CAP.log 248375 mesages sent to URL, 1163154 messages dropped Message Buffer (total:inuse:free) 0001:0000:0001 Buffer Memory: 1024 bytes, Message size: 1024 bytes
Now that the DSP Crash, and HPI capture are up and running lets check the TCP connections. I would recommend checking the filesystem of your Server as well to make sure the HPI log file was created. Don’t be alarmed if you don’t see a DSP crash file on the filesystem of the Server. It won’t actually get created until the crash occurs.
R5#show tcp brief TCB Local Address Foreign Address (state) 24718554 172.16.21.1.55405 SERVER.21 ESTAB 3DC37E44 192.168.21.1.61858 192.168.0.20.2000 ESTAB 21B8A980 192.168.21.1.59226 192.168.0.20.2000 ESTAB 21B8A264 192.168.21.1.61005 192.168.0.20.2000 ESTAB 21B8B09C 192.168.21.1.57360 192.168.60.20.2428 ESTAB 21E60400 172.16.21.1.64903 SERVER.59665 ESTAB 3C3888B0 192.168.21.1.17787 192.168.0.20.2428 ESTAB
TAC Cisco released a BUG ( CSCuo81444 ) fix related to the DSP crashing resource. At the time of this post I’m currently running on IOS version 15.4(3)M.
The following was a recommendation from TAC
After the router has reloaded start off by clearing the logging, and counters. When and if the issue happens again collect the show voice dsp statistic tx-rx three times and show interface and show logging when the DSP is in APP_DNLDING before reloading the router.
Please configure enough logging buffer shown below…
- logging queue-limit 3000
- logging buffered 100000000
Known Fixed Releases
- 15.1(4)M9
- 15.2(4)M6.3
- 15.2(4)M7
- 15.3(3)M3.2
- 15.3(3)M4
- 15.4(2)T1.1
- 15.4(2)T2
- 15.4(2.16)T
- 15.4(3)M
I hope you found this post on DSP Farm Crashing helpful and informative. Be sure to let me know what you think by leaving suggestions, and feedback in the comments section below. You can find out more about these and other articles be checking out recent posts and archives. To learn more about me be sure to check out the About page. And as always thanks again for visiting The Packet.
