01
The Problem
What needed to be understood?
Slow OPS flip over can eventually cause loss of light on the optical system, router interface to go down, and ultimately BGP flaps/reconvergence. In short you can experiences outages. One site in specific experienced 2-5 second transition times. We avoid the interface going down after putting 8 sec hold down timers.. but 5 seconds of black holed traffic is not acceptable in our network.
02
Troubleshooting Approach
Work through the layers.
1. Get a working Och 2. Get a working ethernet circuit (ODU4i) 3. Isolate the OLS; traffic generator on side A and terminal loopback on side B 4. Confirm client side is operational; remove the loopback and test from traffic generator to B side router 5. Test L3 from router to router
03
Outcome
What did the process reveal?
Determined to be a client side issue not on the OLS. From testing the traffic generator you can derive the ms switchover from the following steps: a) 160,276,808 TX frames/s 160,370,535 RX frames/s delta= 106,273 lost frames/s b) we're sending 512 byte frames at 4Gb/s (1% of 400GbE) 4,000,000,000 / (512 x 8) = 976,563 frames/sec c) lost frames/sec / frames/sec 106,273 / 976,563 = 0.108 seconds (switchover time) This proves a fast switchover on one side through the OLS and you can isolate this test with a terminal loopback on the far side.
Full Case Study
1. Och works
2. Circuit works
3. A side (loopback in place): At 1% load on 400G, I sent roughly 4 Gb/s. With 512-byte frames, 106,273 missing frames corresponding to roughly 0.11 seconds, or ~110 ms, of traffic interruption.
B side (traffic generator): At 1% load on 400G, I sent roughly 4Gb/s. With 512-byte frames, 127,979 missing frames correspond to roughly 0.13 seconds or ~131 ms, of traffic interruption.
Now we can determine the issue is a client side connection.
Discussion
Comments