IWETHEY v. 0.3.0 | TODO
1,095 registered users | 0 active users | 0 LpH | Statistics
Login | Create New User
IWETHEY Banner

Welcome to IWETHEY!

New Anyone else seeing weird networking issues with...
Verizon FiOS in Southern California?

This started late Thursday Night or Early Friday Morning.

What is happening is this:

With SSH:
logins are fine.
Do an "ls" in a small directory, returns just fine.
Do an "ls -l" (or any large output), you get the first line to line and half...
ssh session fall off after timeout.
If you keep the ssh network payload less than a 1496MTU (out of a max 1500MTU) with no continuations. It works just fine.
Go over 1496MTU and this happens:
07:51:35.746114 152.63.97.106 > 10.XX.XX.240: icmp: 98.119.XX.XX unreachable - need to frag (mtu 1496) for XX.XX.XX.240.ssh > 98.119.XX.XX.49831: . 0:1448(1448) ack 1 win 11880 <nop,nop,timestamp 3031513704 63904225> (DF) (ttl 56, id 5839, len 1500) (ttl 244, id 48517, len 168)


Similar issue but not the same for HTTP/S on any port:

I see the initial Request come in for a connect, then the SSL request.
My servers respond properly.
Pretty much get NO FURTHER traffic from then on.


We've got multiple large customers opening tickets with us... claiming down times starting on Friday morning. I've got no downtimes and not throughput issues.

Here is something I played with. It looks like its packet sized based issues.

I've been able to trigger it in ICMP as well... but not consistently
ping -I XX.XX.XX.205 -s 1472 98.119.XX.XX

I'm pining with the EXACT size of the 1500MTU. 1472 plus regular headers == 1500
12:46:43.132928 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)

12:46:44.132672 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:45.132579 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:46.131813 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:47.132868 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:48.132831 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:48.186736 152.63.97.102 > 10.50.205.230: icmp: 98.119.XX.XX unreachable - need to frag (mtu 1496) for 10.50.205.230 > 98.119.XX.XX: icmp: echo request (DF) (ttl 55, id 0, len 1500) (ttl 244, id 51932, len 168)
12:46:49.131975 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:50.132900 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:51.135857 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:51.192831 152.63.97.102 > XX.XX.XX.230: icmp: 98.119.XX.XX unreachable - need to frag (mtu 1496) for XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 55, id 0, len 1500) (ttl 244, id 52307, len 168)
12:46:52.136957 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:53.136942 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:54.137060 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:55.136044 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:56.136908 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:57.136049 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:58.136110 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)
12:46:59.137047 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1500)


But if I make it one more byte to big.
ping -I XX.XX.XX.205 -s 1473 98.119.XX.XX

12:49:12.599154 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (frag 50193:1480@0+) (ttl 62, len 1500)
12:49:12.599157 XX.XX.XX.205 > 98.119.XX.XX: icmp (frag 50193:1@1480) (ttl 62, len 21)
12:49:12.671611 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (frag 63953:1480@0+) (ttl 54, len 1500)
12:49:12.672577 98.119.XX.XX > XX.XX.XX.205: icmp (frag 63953:1@1480) (ttl 54, len 21)
12:49:13.598942 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (frag 50194:1480@0+) (ttl 62, len 1500)
12:49:13.598947 XX.XX.XX.205 > 98.119.XX.XX: icmp (frag 50194:1@1480) (ttl 62, len 21)
12:49:13.671569 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (frag 63954:1480@0+) (ttl 54, len 1500)
12:49:13.672530 98.119.XX.XX > XX.XX.XX.205: icmp (frag 63954:1@1480) (ttl 54, len 21)
12:49:14.600058 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (frag 50195:1480@0+) (ttl 62, len 1500)
12:49:14.600062 XX.XX.XX.205 > 98.119.XX.XX: icmp (frag 50195:1@1480) (ttl 62, len 21)
12:49:14.658594 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (frag 63955:1480@0+) (ttl 54, len 1500)
12:49:14.658605 98.119.XX.XX > XX.XX.XX.205: icmp (frag 63955:1@1480) (ttl 54, len 21)
12:49:15.600015 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (frag 50196:1480@0+) (ttl 62, len 1500)
12:49:15.600019 XX.XX.XX.205 > 98.119.XX.XX: icmp (frag 50196:1@1480) (ttl 62, len 21)
12:49:15.658587 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (frag 63956:1480@0+) (ttl 54, len 1500)
12:49:15.658599 98.119.XX.XX > XX.XX.XX.205: icmp (frag 63956:1@1480) (ttl 54, len 21)


or 8 bytes smaller.
ping -I XX.XX.XX.205 -s 1466 98.119.XX.XX

12:50:01.849470 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (ttl 54, id 63957, len 1494)
12:50:02.790456 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1494)
12:50:02.849645 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (ttl 54, id 63958, len 1494)
12:50:03.789621 XX.XX.XX.205 > 98.119.XX.XX: icmp: echo request (DF) (ttl 62, id 0, len 1494)
12:50:03.848247 98.119.XX.XX > XX.XX.XX.205: icmp: echo reply (ttl 54, id 63959, len 1494)

--
greg@gregfolkert.net
PGP key 1024D/B524687C 2003-08-05
Fingerprint: E1D3 E3D7 5850 957E FED0 2B3A ED66 6971 B524 687C
New As of 3 hours ago...
Verizon fixed the problem.

No acknowledgement of the problem, not conveyance of the fix, not even notice it has been fixed.

It was quite wide spread as well, Southern CA, Parts of AZ and Nevada. Eastern Seaboard and a few areas in the midwest.
--
greg@gregfolkert.net
PGP key 1024D/B524687C 2003-08-05
Fingerprint: E1D3 E3D7 5850 957E FED0 2B3A ED66 6971 B524 687C
New Would you expect more from Verizon?
New No, not really.
But it would be nice to know what they screwed up.

I think its doing with their Net Neutrality issues they are stirring up.
--
greg@gregfolkert.net
PGP key 1024D/B524687C 2003-08-05
Fingerprint: E1D3 E3D7 5850 957E FED0 2B3A ED66 6971 B524 687C
     Anyone else seeing weird networking issues with... - (folkert) - (3)
         As of 3 hours ago... - (folkert) - (2)
             Would you expect more from Verizon? -NT - (mmoffitt) - (1)
                 No, not really. - (folkert)

I wonder how long before this screw job ends up as my boss?
151 ms