Good Morning! It’s amazing what just a couple hours sleep will do for you :)
We hope everyone had a great DEF CON 20 Friday night. Great turn out at all the parties, with some great entertainment (DJ’s, bands, etc). Too many for any one person to see all of them (kind of like Disneyland, right?)
Everyone’s going to be asking about DCTV - right, let’s get on with that. Several of us were up late (me, Mac, Richard) working on ideas to get things fixed & back to working order. I think we finalized on Plan Echo(E). We still have Plan Foxtrot(F) in our back pocket. What was wrong? Combination of things. Primary issue was a codec that was mostly-standard, but slightly not. Couple that with newer versions of software that didn’t handle the “slightly not” part right. We found the right combination of software and command-line flags, and at 12:30am last night declared (very quietly, so we didn’t jinx it) victory - at least with our proof-of-concept we setup in the NOC. We’ll re-deploy equipment in a few minutes and verify our hypothesis in production!
Again, we apologize that things weren’t running well yesterday. This has been our #1 priority to resolve since yesterday morning and appreciate your patience. (I know I sound calm now, but I was such a stress-ball last night!!).
Other than that, we are having a lot of people using wifi, and great traffic. I think we’re going to have to upgrade next year. Our controller is running out of memory throughout the day. Rukbat has been amazingly diligent at sitting quietly in the NOC, all day long, clearing memory, keeping it running as best he can. Can’t say enough good things about Rukbat, seriously - I should do an entire post about this guy! He’s bloody incredible! But meanwhile, if you do see intermittent loss of connectivity, either wait a minute or two (I know, patience is hard when you’re trying to pull up a webpage with someone standing next to you waiting on it) - or email us (noc-at-defconnetworking-dot-org) and let us know. A couple people did yesterday, it’s helpful, so thank you guys.
Looks like all afternoon ya’ll were using some massive bandwidth. Here, have a look…