User:Sfiggins/Broadworks Controlled Registration - Revision history

Sfiggins: /* Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. */

2019-05-27T01:42:25Z

Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.

Show changes

Sfiggins: /* SBC CPU with ACL applied */

2019-05-27T01:20:32Z

SBC CPU with ACL applied

Sfiggins: /* Controlled Registration / Removal of ACL */

2019-05-27T01:20:15Z

Controlled Registration / Removal of ACL

Sfiggins: /* Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. */

2019-05-27T01:17:32Z

Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.

Sfiggins: /* Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. */

2019-05-27T01:10:32Z

Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.

Sfiggins: /* Controlled Registration / Removal of ACL */

2019-05-27T01:03:19Z

Controlled Registration / Removal of ACL

Sfiggins: /* Verify om AS1/2 that the RegistrationsPerMinute has dropped */

2019-05-27T01:02:57Z

Verify om AS1/2 that the RegistrationsPerMinute has dropped

Sfiggins: /* Apply ACL on sw4-inverness-co */

2019-05-27T01:01:59Z

Apply ACL on sw4-inverness-co

Sfiggins at 00:52, 27 May 2019

2019-05-27T00:52:50Z

Sfiggins: /* Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. */

2019-05-27T00:36:44Z

Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.

@@ Line 368: / Line 368: @@
 With the ACL applied, there should be lower little SIP traffic, and the CPU will be low.
 Denver3820# show processes cpu
 Task Name        Task Id Pri Status          Total CPU   Avg   Now

← Older revision		Revision as of 01:20, 27 May 2019
Line 363:		Line 363:

	When the '''RegistrationsPerMinute''' has dropped to a substantially low number, like '''39''' shown above, you can start to remove lines from the ACL.		When the '''RegistrationsPerMinute''' has dropped to a substantially low number, like '''39''' shown above, you can start to remove lines from the ACL.
		+
		+	=== SBC CPU with ACL applied ===
		+
		+	With the ACL applied, there should be lower little SIP traffic, and the CPU will be low.
		+
		+	Denver3820# show processes cpu
		+	Task Name Task Id Pri Status Total CPU Avg Now
		+	-------------- -------- --- ---------- -------------- ----- -----
		+	tSipd 1c8c8144 80 READY 47:17:26.151 8.0 2.5
		+	tFlowGdTmr 0d43b030 62 PEND+T 4:06:24.624 0.6 0.6
		+	tNpDmaRx 0d399220 61 READY 2:41:14.421 0.4 0.3
		+	tNpDmaTx 0d3e59f0 60 PEND 3:13:38.455 0.5 0.1
		+	tMbcd 0347a64c 78 PEND+T 1:00:44.814 0.1 0.0
		+	tIpFrag 17218adc 60 PEND 42:58.989 0.1 0.0
		+	tNpwbNpmRx 0d3999c4 60 READY 29:48.228 0.0 0.0
		+	tAlarm 1cbfc878 60 DELAY 26:58.348 0.0 0.0
		+	BusM A 033460b4 100 READY 25:18.960 0.0 0.0
		+	tNetTask 03350b10 50 PEND 24:48.192 0.0 0.0
		+	tArpMgr 0d45388c 61 PEND 9:44.400 0.0 0.0
		+	tAtcpd 03468c84 75 PEND+T 1:07.465 0.0 0.0
		+	nPCSL_timer 03436cb4 100 DELAY 1:03.450 0.0 0.0
		+	tSysmand 0d19dc50 75 PEND+T 1:02.962 0.0 0.0
		+	tTaskCheck 0342a664 100 DELAY 44.914 0.0 0.0
		+	tAlgd 1c788d2c 80 PEND+T 40.815 0.0 0.0
		+	-------------- -------- --- ---------- -------------- ----- -----
		+	Applications 60:38:36.079 10.3
		+	System 587:41:47 4.1
		+	</pre>

	=== Controlled Registration / Removal of ACL ===		=== Controlled Registration / Removal of ACL ===

← Older revision		Revision as of 01:17, 27 May 2019
Line 51:		Line 51:

	If this number is jumping around from from 6,000-18,000, the platform will not be able to recover itself, and you need to proceed with the rest of the document. If this number is steadily decreasing, you can monitor it to see if it will recover on its own.		If this number is jumping around from from 6,000-18,000, the platform will not be able to recover itself, and you need to proceed with the rest of the document. If this number is steadily decreasing, you can monitor it to see if it will recover on its own.
		+
		+	=== Log into the SBC and check the CPU ===
		+
		+	Log into castrum again, then telnet into the SBC.
		+
		+	<pre>
		+	[sfiggins@castrum ~]$ telnet 64.1.10.132
		+	Trying 64.1.10.132...
		+	Connected to 64.1.10.132.
		+	Escape character is '^]'.
		+
		+	Password:
		+	Denver3820> en
		+	Password:
		+	Denver3820#
		+	</pre>
		+
		+	Look a the CPU.
		+
		+	<pre>
		+	Denver3820# show processes cpu
		+	Task Name Task Id Pri Status Total CPU Avg Now
		+	-------------- -------- --- ---------- -------------- ----- -----
		+	tSipd 1c8c8144 80 READY 47:15:13.431 8.0 10.3
		+	tNpDmaTx 0d3e59f0 60 PEND 3:13:26.747 0.5 1.0
		+	tFlowGdTmr 0d43b030 62 PEND+T 4:06:15.324 0.6 0.6
		+	tNpDmaRx 0d399220 61 READY 2:41:07.554 0.4 0.5
		+	tMbcd 0347a64c 78 PEND+T 1:00:41.640 0.1 0.1
		+	tIpFrag 17218adc 60 PEND 42:58.169 0.1 0.0
		+	tNpwbNpmRx 0d3999c4 60 READY 29:47.101 0.0 0.0
		+	tAlarm 1cbfc878 60 DELAY 26:57.329 0.0 0.0
		+	BusM A 033460b4 100 READY 25:18.067 0.0 0.0
		+	tNetTask 03350b10 50 PEND 24:47.240 0.0 0.0
		+	tArpMgr 0d45388c 61 PEND 9:44.128 0.0 0.0
		+	tAtcpd 03468c84 75 PEND+T 1:07.423 0.0 0.0
		+	nPCSL_timer 03436cb4 100 DELAY 1:03.407 0.0 0.0
		+	tSysmand 0d19dc50 75 PEND+T 1:02.914 0.0 0.0
		+	tTaskCheck 0342a664 100 DELAY 44.885 0.0 0.0
		+	tAlgd 1c788d2c 80 PEND+T 40.789 0.0 0.0
		+	-------------- -------- --- ---------- -------------- ----- -----
		+	Applications 60:35:47.922 10.3
		+	System 587:19:12 13.3
		+	</pre>
		+
		+	The SBC should be between 5 and 15% CPU. If this is stable, the SBC will not need further remediation. It is it higher, check it again after the ACL is in place. Without traffic, the CPU on the SBC should be very low.
		+
		+	Additional commands that are of interest:
		+
		+	<pre>
		+	Denver3820# show sipd realms
		+	11:57:51-57
		+	----- Inbound ----- ---- Outbound ----- -- Latency -- Max
		+	Realm Active Rate ConEx Active Rate ConEx Avg Max Burst
		+	VGtoBSoft I 11 0.4 0 0 0.0 0 0.000 0.000 9
		+	VGtoBSoft_private I 0 0.0 0 11 0.4 0 2.042 16.442 9
		+	bw-as-access I 0 0.0 0 0 0.0 0 0.084 0.590 66
		+	bw-as-core I 0 0.0 0 0 0.0 0 11.710 29.464 66
		+	ils-t7k-access I 0 0.0 0 0 0.0 0 0.000 0.000 0
		+	ils-t7k-core I 0 0.0 0 0 0.0 0 0.000 0.000 0
		+	soho-o365-private I 0 0.0 0 0 0.0 0 0.000 0.000 0
		+	soho-o365-public I 0 0.0 0 0 0.0 0 0.000 0.000 0
		+	soho-um-private I 0 0.0 0 0 0.0 0 0.000 0.000 0
		+	soho-um-public I 0 0.0 0 0 0.0 0 0.000 0.000 0
		+	</pre>
		+
		+	This will show which realms are in use. It does not seem to track registrations. The above no calls involving the phones.
		+
		+	<pre>
		+	Denver3820# show sipd realms bw-as-access
		+	11:58:24-30
		+	Realm bw-as-access() [In Service]
		+	-- Period -- -------- Lifetime --------
		+	Active High Total Total PerMax High
		+	Inbound Sessions 0 0 0 259720 44 52
		+	Rate Exceeded - - 0 0 0 -
		+	Num Exceeded - - 0 0 0 -
		+	Burst Rate 0 0 0 0 0 12
		+	Reg Rate Exceeded - - 0 0 0 -
		+	Reg Burst Rate 0 0 0 0 0 0
		+	Outbound Sessions 0 0 0 496348 102 113
		+	Rate Exceeded - - 0 0 0 -
		+	Num Exceeded - - 0 0 0 -
		+	Burst Rate 0 0 0 0 0 63
		+	Reg Rate Exceeded - - 0 0 0 -
		+	Out of Service - - 0 0 0 -
		+	Trans Timeout 0 0 0 0 0 0
		+	Requests Sent - - 9 10193974 1223 -
		+	Requests Complete - - 8 8888161 1197 -
		+	Seizure - - 0 508581 103 -
		+	Answer - - 0 137077 68 -
		+	ASR Exceeded - - 0 0 0 -
		+	Requests Received - - 6818 274595943 17291 -
		+	QoS Major Exceeded - - 0 0 0 -
		+	QoS Critical Exceeded - - 0 0 0 -
		+	Latency=0.609; max=2.482
		+	QoS R-Factor Avg=0.00; max=0.00
		+	</pre>
		+
		+	You can see that there is not much going on in these counters.

	=== Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. ===		=== Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. ===

← Older revision		Revision as of 01:10, 27 May 2019
Line 1:		Line 1:

		+
		+	=== Verify RegistrationsPerMinute on AS1/2 ===
		+
		+	If the '''RegistrationsPerMinute''' count on the active Application server is greater than a couple hundred, the platform will need help recovering.
		+
		+	Log into castrum.risebroadband.com, and then ssh to 172.16.18.3 or 10.1.75.181, depending on which AS is active.
		+
		+	<pre>
		+	$ ssh castrum.risebroadband.com
		+	[sfiggins@castrum ~]$ ssh 172.16.18.3 -l bwadmin
		+	bwadmin@172.16.18.3's password:
		+
		+	bwadmin@as1.jabvoice.net$ bwcli
		+
		+
		+	======================================================================
		+	BroadWorks Command Line Interface
		+	Type HELP for more information
		+	======================================================================
		+	Reading initial CLI command file...
		+
		+	AS_CLI>
		+	</pre>
		+
		+	Then run the '''qcurrent''' command to see the current state of the queue.
		+
		+	<pre>
		+	AS_CLI> qcurrent
		+	May 24, 2019 =====================================
		+	NbOfActiveCalls 2.0
		+	SIPSetupSignalDelay 16.0
		+	SIPAnswerSignalDelay 2.0
		+	RegistrationsPerMinute 12821.0
		+	SIPMsgRetryToNE::10.1.75.182 0.0
		+	SIPMsgRetryToNE::172.16.18.4 0.0
		+	SIPMsgRetryToNE::64.1.8.229 0.0
		+	SIPMsgRetryToNE::172.16.18.5 0.0
		+	SIPMsgRetryToNE::172.16.18.148 75.0
		+	SIPMsgRetryToNE::10.1.75.183 0.0
		+	SIPMsgRetryToNE::172.16.18.3 0.0
		+	SIPMsgRetryToNE::10.1.75.181 83.0
		+	SIPMsgRetryToNE::172.16.18.130 0.0
		+	SIPMsgRetryPercentToOther 91.0
		+	MGCPDialtoneDelay 0.0
		+	MGCPSetupSignalDelay 0.0
		+	MGCPAnswerSignalDelay 0.0
		+	MGCPMessageRetryPercent 0.0
		+	CallsPerSecond 0.0
		+	</pre>
		+
		+	If this number is jumping around from from 6,000-18,000, the platform will not be able to recover itself, and you need to proceed with the rest of the document. If this number is steadily decreasing, you can monitor it to see if it will recover on its own.
		+
	=== Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. ===		=== Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts. ===

@@ Line 223: / Line 223: @@
 </pre>
-Enter the following commands, 5-10 at a time, while monitoring the per-minute registration rate
+Enter the following commands, '''5-10 at a time''', while monitoring the per-minute registration rate
 on AS1/2.  If the rate raises, ensure that it is dropping back to less than 200 before continuing
 with the next set of commands.  Total time should be less than 10 minutes.

@@ Line 157: / Line 157: @@
 === Verify om AS1/2 that the RegistrationsPerMinute has dropped ===
-While still logged into AS1 or AS2, verify that the registrationsPerMinute count has dropped.
+While still logged into AS1 or AS2, verify that the '''RegistrationsPerMinute''' count has dropped.
 <pre>
@@ Line 211: / Line 211: @@
 If '''Enterprise Support''' team is in Loveland, and '''72.19.129.42''' is not in the ACL, they can check to see if their phones are working.  If it is in the ACL, you can remove that line, and have '''Enterprise Support''' retest.
-When this has dropped to a substantially low number, like '''39''' shown above, you can start to remove lines from the ACL.
+When the '''RegistrationsPerMinute''' has dropped to a substantially low number, like '''39''' shown above, you can start to remove lines from the ACL.
 === Controlled Registration / Removal of ACL ===

@@ Line 146: / Line 146: @@
 </pre>
-Then paste in the ACL generated from the cn10-inverness-co.
+'''Paste in the ACL generated from the cn10-inverness-co.'''
 After it is pasted in, run the following command:
@@ Line 154: / Line 154: @@
 sw4-inverness-co#
 </pre>
 === Controlled Registration / Removal of ACL ===

@@ Line 8: / Line 8: @@
     and dst 64.1.10.130 > /tmp/sbc-$DATE.pcap
-# echo "ip access-list extended SCB-Security-$DATE"; cat /tmp/sbc-$DATE.pcap \
+# echo "ip access-list extended SBC-Security-$DATE"; cat /tmp/sbc-$DATE.pcap \
     | awk '{print $3}' | perl -npe 's/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*$/$1/g' | sort \
     | uniq -c | sort -n | tail -n 110 | awk '{print " deny ip host "$2" host 64.1.10.130"}' \
-    | sort -V  | uniq; echo " permit ip any any"; echo "interface vlan80"; \
+    | sort -V  | uniq; echo " permit ip any any";echo '!'; echo "interface vlan80"; \
-    echo " ip access-class SCB-Security-$DATE out"; rm -f /tmp/sbc-$DATE.pcap
+    echo " ip access-group SBC-Security-$DATE out"; rm -f /tmp/sbc-$DATE.pcap
 </pre>
@@ Line 19: / Line 19: @@
 <pre>
-ip access-list extended SCB-Security-2019-05-26-1728
+ip access-list extended SBC-Security-2019-05-26-1728
   deny ip host 12.44.24.2 host 64.1.10.130
   deny ip host 12.70.162.58 host 64.1.10.130
@@ Line 131: / Line 131: @@
   deny ip host 216.228.69.74 host 64.1.10.130
   permit ip any any
 interface vlan80
-  ip access-class SCB-Security-2019-05-26-1728 out
+  ip access-group SCB-Security-2019-05-26-1728 out
 </pre>