- To: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>, Arthur A Person <aap1@xxxxxxx>, "_NCEP.List.pmb-dataflow" <ncep.list.pmb-dataflow@xxxxxxxx>
- Subject: Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.
- From: Pete Pokrandt <poker@xxxxxxxxxxxx>
- Date: Fri, 19 Feb 2016 19:52:10 +0000
- Authentication-results: albany.edu; dkim=none (message not signed) header.d=none; albany.edu; dmarc=none action=none header.from=aos.wisc.edu;
- Spamdiagnosticmetadata: NSPM
- Spamdiagnosticoutput: 1:23
An update - just before noon CST I updated to the newest ldm (6.12.14) and also
switched my CONDUIT request sites so I'm requesting from idd.unidata.ucar.edu
and ncepldm4.woc.noaa.gov (both in Boulder, I believe) and commented out the
request to conduit.ncep.noaa.gov (in Washington DC?)
Latencies are almost immediately better (see attached gif) and the 18 UTC NAM
seems to be coming in fine. Guess I'm good for now, though me feeding CONDUIT
from idd.unidata.ucar.edu doesn't seem ideal.. Would be better to figure out
why I'm getting such latencies from the two noaa sites.
Some mtr data:
idd.aos.wisc.edu to idd.unidata.ucar.edu:
My traceroute [v0.75]
idd.aos.wisc.edu (0.0.0.0)
Fri Feb 19 13:48:54 2016
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host
Loss% Snt Last Avg Best Wrst StDev
1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu
0.0% 6291 6.9 0.9 0.5 49.6 3.7
2. internet2-ord-600w-100G.net.wisc.edu
0.0% 6291 18.2 18.1 18.0 70.9 3.8
3. ae0.3454.core-l3.frgp.net
0.0% 6291 40.2 40.2 39.9 91.3 4.0
4. tcom-gs-1-vl873.ucar.edu
0.0% 6291 41.4 41.1 40.8 160.8 6.6
5. flra.ucar.edu
0.0% 6291 41.5 45.7 40.9 406.8 22.9
6. idd.unidata.ucar.edu
0.2% 6290 41.8 41.1 40.9 106.1 4.8
idd.aos.wisc.edu to ncepldm4.woc.noaa.gov
My traceroute [v0.75]
idd.aos.wisc.edu (0.0.0.0) Fri
Feb 19 13:49:42 2016
Keys: Help Display mode Restart statistics Order of fields quit
Packets
Pings
Host Loss% Snt Last
Avg Best Wrst StDev
1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu 0.0% 7014 1.3
0.9 0.6 50.0 3.8
2. internet2-ord-600w-100G.net.wisc.edu 0.0% 7014 18.7
18.2 18.1 67.9 3.7
3. ae0.3454.core-l3.frgp.net 0.0% 7014 54.7
40.3 40.0 90.9 4.0
4. noaa-i2.frgp.net 0.0% 7014 50.0
41.4 40.8 106.0 5.7
5. 2001-mlx8-eth-1-2.boulder.noaa.gov 0.0% 7014 58.3
42.2 40.8 107.8 5.6
6. mdf-rtr-6.boulder.noaa.gov 0.0% 7014 56.9
42.3 40.9 107.0 5.7
7. ???
idd.aos.wisc.edu to conduit.ncep.noaa.gov
My traceroute [v0.75]
idd.aos.wisc.edu (0.0.0.0) Fri
Feb 19 13:49:18 2016
Keys: Help Display mode Restart statistics Order of fields quit
Packets
Pings
Host Loss% Snt Last
Avg Best Wrst StDev
1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu 0.0% 7031 1.6
0.8 0.6 50.6 3.6
2. internet2-ord-600w-100G.net.wisc.edu 0.0% 7031 18.2
18.1 18.0 68.7 3.6
3. et-10-0-0.107.rtr.clev.net.internet2.edu 0.0% 7031 27.9
27.8 27.6 76.4 3.8
4. et-11-3-0-1276.clpk-core.maxgigapop.net 0.0% 7031 37.3
37.3 37.1 86.1 4.3
5. noaa-i2.demarc.maxgigapop.net 0.0% 7031 39.0
40.6 37.3 373.8 20.6
6. 140.90.111.36 0.0% 7031 37.6
53.5 37.4 468.2 34.2
7. 140.90.76.69 0.0% 7031 37.8
38.6 37.5 130.1 6.6
8. ???
Pete
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx
________________________________
From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> on
behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
Sent: Friday, February 19, 2016 11:20 AM
To: Carissa Klemmer - NOAA Federal; Arthur A Person; _NCEP.List.pmb-dataflow
Cc: Bentley, Alicia M; Daes Support; support-conduit@xxxxxxxxxxxxxxxx; Michael
Schmidt
Subject: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu
starting the last day or two.
All,
Not sure if this is on my end or somewhere upstream, but the last several runs
my CONDUIT latencies have been getting huge to the point where we are losing
data.
I did stop my ldm the other day to add in an alternate feed for Gilbert at
allisonhous.com, not sure if that pushed me over a bandwidth limit, or by
reconnecting we got hooked up to a different remote ldm, or taking a different
path, that shot the latencies up.
Seems to be really only CONDUIT, none of our other feeds show this kind of
latency.
Still looking into things locally, but wanted make people aware. I just
rebooted idd.aos.wisc.edu, will see if that helps at all.
Here's an ldmping and traceroute from idd.aos.wisc.edu to conduit.ncep.noaa.gov.
[ldm@idd ~]$ ldmping conduit.ncep.noaa.gov
Feb 19 17:16:08 INFO: State Elapsed Port Remote_Host
rpc_stat
Feb 19 17:16:08 INFO: Resolving conduit.ncep.noaa.gov to 140.90.101.42 took
0.00486 seconds
Feb 19 17:16:08 INFO: RESPONDING 0.115499 388 conduit.ncep.noaa.gov
traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte
packets
1 r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3) 0.760 ms
0.954 ms 0.991 ms
2 internet2-ord-600w-100G.net.wisc.edu (144.92.254.229) 18.119 ms 18.123 ms
18.107 ms
3 et-10-0-0.107.rtr.clev.net.internet2.edu (198.71.45.9) 27.836 ms 27.852
ms 27.838 ms
4 et-11-3-0-1276.clpk-core.maxgigapop.net (206.196.177.4) 37.363 ms 37.363
ms 37.345 ms
5 noaa-i2.demarc.maxgigapop.net (206.196.177.118) 38.051 ms 38.254 ms
38.401 ms
6 140.90.111.36 (140.90.111.36) 118.042 ms 118.412 ms 118.529 ms
7 140.90.76.69 (140.90.76.69) 41.764 ms 40.343 ms 40.500 ms
8 * * *
9 * * *
10 * * *
Similarly to ncepldm
[ldm@idd ~]$ ldmping ncepldm4.woc.noaa.gov
Feb 19 17:18:40 INFO: State Elapsed Port Remote_Host
rpc_stat
Feb 19 17:18:40 INFO: Resolving ncepldm4.woc.noaa.gov to 140.172.17.205 took
0.001599 seconds
Feb 19 17:18:40 INFO: RESPONDING 0.088901 388 ncepldm4.woc.noaa.gov
^C
[ldm@idd ~]$ traceroute ncepldm4.woc.noaa.gov
traceroute to ncepldm4.woc.noaa.gov (140.172.17.205), 30 hops max, 60 byte
packets
1 r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3) 0.730 ms
0.831 ms 0.876 ms
2 internet2-ord-600w-100G.net.wisc.edu (144.92.254.229) 18.092 ms 18.092 ms
18.080 ms
3 ae0.3454.core-l3.frgp.net (192.43.217.223) 40.196 ms 40.226 ms 40.256 ms
4 noaa-i2.frgp.net (128.117.243.11) 40.970 ms 41.012 ms 40.996 ms
5 2001-mlx8-eth-1-2.boulder.noaa.gov (140.172.2.18) 42.780 ms 42.778 ms
42.764 ms
6 mdf-rtr-6.boulder.noaa.gov (140.172.6.251) 40.869 ms 40.922 ms 40.946 ms
7 * * *
8 * * *
Pete
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx
Attachment:
Screenshot at 2016-02-19 13:46:11.png
Description: Screenshot at 2016-02-19 13:46:11.png
- Follow-Ups:
- Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.
- From: Patrick L. Francis
- Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.
- References: