DefconNetworking.org: Home

DEF CON 33 - Here’s some data

This year, sparky and mac wanted to start sharing a bit more of what we do and how things go behind the scene. In that sense, I am going to throw at you a lot of boring numbers, charts… and some fun stuff we did this year.

First we do the work

Like mac mentioned on his post, this year we knew beforehand that we would have less time to setup everything. So we had to be creative and come up with ways to reduce our setup time, and one of those ideas was to use wireless bridges to significantly reduce the number of physical drops we had to run all over the convention center.

On Monday, we got together in a hotel space and worked as a team to pre-stage as much of the equipment as possible, that included the 62 wireless bridges we were going to deploy all over LVCC.

We were excited to see how well those wireless bridges would work… and as you can see on the graphs below, they worked surprisingly well, considering the busy environment.

And yes, we heard a good number of folks were complaining about the wireless network. But I wanted to take this moment to clarify that as far as data goes, the wireless network was actually rock solid, low latency and all that good stuff. We did have one issue Saturday afternoon, where the DHCP decided to take a nap and made it look like the DefCon wireless networks were offline. That lasted about 20 minutes before everything was up and running again. So when folks rightfully complained about “the wifi” on social media, read our internet pipes, and not the wireless itself. Our friends that take care of the wireless networks felt the heat while doing their best, so I wanted to clear that up. Give them some love, they did an amazing job.

The Woppers

That’s what we called those wireless bridges internally, “woppers”.

As you can see below, our average “last mile” latency, which is basically latency between the wopper and the other side of the wireless network, was below 20ms, with sub-10ms on a good chunk of them. For this kind of busy environment, that is pretty solid. The “reds” you see are packet losses, and we were monitoring these closely throughout the day. When we saw them, we went to physically check on the wopper, and 100% of the time was due to people moving them under tables or even under other objects/bags. For next year that is something we plan to address by sticking them to high poles or similar.

Then we have fun

This year we deployed some extra monitoring around the LVCC to see what you all were up to… and you didn’t disappoint us! Good job! We were using Nzyme for this.

Good news first, we didn’t see any significant/relevant de-auth attacks on the wireless network. As you can see on the graph below, there is no “spike” in de-auths, other than the expected from normal activities. And to be honest, if you are connect to the WPA3-only network, sans an RF blast on all frequencies, you are safe from those kind of attacks. I am not saying people didn’t try, it was just not enough to cause any trouble.

This is DEF CON

When they say out there this is a hostile environment… this is what they mean. We of course had folks blasting around all kinds of SSIDs, like “Comcast”, “Ford-5G”, airport lobby, stores… you name it. As you probably understands, your phone WILL automatically connect to those networks if you ever connected to them and left the “Remember Network” box checked… which is no bueno.

And by the way, trilateration works and we could pinpoint where those funny people were. ;)

Someone (or multiple people?) also dropped ESP32s all over the LVCC. I did some reverse engineering on the firmware to see what was going on. They were basically broadcasting a network with the same name as DefCon-Open, counting how many users connected to it, and they could also setup a fake captive portal with a password form that redirected to a page that played Rick Astley’s famous song.

Lastly, here are some other interesting things we were able to see:

Rogue devices:

ESP32 Marauders (not unique devices, but times we saw them advertising): 11486
Unique Pwnagotchi: 23
Unique Flipper Zero Evil Portal: 12
Unique Wifi Pineapples: 34

Fake Doppelgangers of our Networks:

Networks with same name as Defcon Secure Networks but disabled encryption (Unique sources): 10
Networks with similar name as Defcon networks (Unique sources): 5

Other stats:

UAVs (drones) sending beacons around LVCC West: 86
Management Frames processed in bytes: 26,360,159,156 (26GB)
Peak number of devices advertising networks around LVCC: 6023

Interesting SSIDs

Before I list them, a few remarks:

• Yes, there is a Dunkin' Donuts at the LVCC. No, they don't have a free wifi network without a captive portal!
• Our open network is DefCon-Open, not DEFCON OPEN... I recommend you connect to neither...
• Good old Rick Roll is back this year...

SpaceLasers
iPhone
Dunkin' Donuts Guest
It Hurts When IP
You shall not connect
IG Labs Regional Airport_5G
FakeNews
IG Labs Regional Airport
01 Never gonna give you up
02 Never gonna let you down
03 Never gonna run around
04 and desert you
08 and hurt you
07 Never gonna tell a lie
05 Never gonna make you cry
06 Never gonna say goodbye
"Nock, nock, Luke."
FBI Van #4
House LANister
FBI-surveillance-van
DEFCON OPEN
FBI Surveillance Van
NSA_Surveillance
NSAsurveillanceVan01

Even with all the challenges, it was a good year. And I can’t wait for DEF CON 34!!

– strange

Welp, another year, another DEF CON come and gone.

On the whole, it was a big success for the DEF CON NOC. We tried out a few new things that went well. The biggest one was some wireless bridges. For reasons, we had a shortened on site setup time, so we really had to cut down our install time. Mostly this meant cutting down on the number of floor wired drops that we could provide or providing them in a much fast way. We opted for the second option by providing some wireless bridges. We’ll put up a post later with the specifics of these, but all in all, we think they did the job.

We of course ran into a few hiccups, some self inflicted. As usual, the bandwidth crunch was a thing on the first day. Apparently, DEF CON is the time for everyone to download all of the patches when arriving onsite, because, well, reasons. Luckily, bossman DT paid for the bandwidth upgrade, doubling our overall commit from 500Mbps to 1Gbps. On the whole, we feel it worked out over the rest of the weekend. We’re gonna start there next time.

Sometimes, you forget the little things, like making sure you have enough log retention to be able to count number of devices and what not, and then you forget the little things like test after configurations and updates. And sometimes, you then have to have appointments that just don’t work well with making changes. Sometimes. Well, completely unrelated, Saturday there was a DHCP outage. It was a quick fix, but still took a bit to respond because, well sometimes.

Lastly, the third system wide thing was running out of source states on the firewall. No matter how much you test blasting the thing, there’s always the scenarios you hit when going into production. Again, luckily, it was a quick fix, but still lessoned learned.

Now, to the numbers. To give you an insight into what we did, here’s some of the normal usage and build out stats:

If you give the bandwidth, people will come. We’re looking at making sure to keep or increase that in the budget for next year. But, just uh, try to patch before getting here. We do what we can, but still recommend against unpatched systems out there.

Expect a follow up on the wireless stats. We’re getting more and more data out of those, just to see what people are playing with.

Until then, thanks for using the network. And thanks for being understanding when there are issues. We know how critical it is to your experience here, and aim to provide the best we can with what we got. It’s always a balance, but it’s good to see the overall numbers increasing every year, and expect to see more and more good things in the future.

Have fun, kids,

– sparky and mac

Another DEF CON is in the books.

Overall, it was a bit of a dip. I think the change of things lead to a lot of unknowns, and it shows in the stats:

• 25% less interwebz bits
• 15% less registered users on wifi
• 10% less concurrent users on wifi

For full information, see the slides.

A couple of incidents to follow up on, but probably the most impactful is to the core. Our firewall is giving up the ghost. We bought this one back in 2014, so 10 years seems to be a more than reasonable time of service. We will be sure to give it a proper Viking funeral and send it across the Rainbow Bridge.

(Planned) New gear this year:

• core switch
• dozen more edge switches
• oh, and the whole freakin’ facility
  • including 650 Access Points

For a new space, everything went exceptionally well. The DEF CON NOC team completed the their work…not only did they complete it…they did it faster, and more efficiently than we have ever. These people are simply the best of the best. The team work, the helping hands, the consistent team spirit in the face of staggering pressure and odds has been out standing.

Thanks to the entire NOC crew: miked, c7five, booger, deadication, dp1i, crv, jon2, wish, strange, k4tn155, toph, commiebstrd, duffguy, tater, meibo, and johntitor. And a special thanks to NOC lead emeritus and still active NOC goon, effffn!

The operations group at DC should also be commended for putting on what could be arguably the best DefCon ever! Thanks to DT, Janet, Nikita, Mo and GAG LAB Department Leads, QM, every other goon, and to all of you.

To Cox, thank you guys for being there. Your dedication to getting it done is not only critical to our success, but building new relationships with ground crews is hard and your guys were always rad!

Thanks to the LVCVA for hosting and working with us. We love the site and look forward to many DEF CONs to come.

Thanks for all the lulz, and see ya next year,

sparky and mac

[NOTE: So, apparently, we missed updating the website last year, so this is done a bit in arears. Meh –mac.]

Another DEF CON is in the books.

Overall, we saw more usage than any previous year:

• 15% more interwebz bits
• 30% more registered users on wifi
• 25% more concurrent users on wifi

And we saw the usual amount of issues. Nothing new there.

For full information, see the slides.

Thanks to the entire NOC crew: miked, c7five, booger, deadication, dp1i, crv, jon2, wish, strange, musa, toph, commiebstrd, and johntitor.

Thanks to DT and the DEF CON staff, all of the other goons, the inhumans, and to all of you. You’re why we do this, and we’re happy to make this happen.

And especially thanks to our hosting partners: Phil, Harry, Mable, and the whole Cesars IT and Encore crews for going above and beyond to help us make this happen.

As a last note, we have to announce a change in leadership. effffn is trying to get some of his life back, and so is stepping back. #sparky, who’s been with the NOC the longest, is stepping up and stepping in.

I welcome our new Canadian overlord!

Thanks for all the lulz, and see ya next year,

efffn and mac

30 years of DEF CON! It was great times.

From a NOC perspective it was a good year. Mostly thanks to Mac, who made a few trips pre-con to Vegas to have some equipment staged.

Overall everything worked. Wi-Fi coverage at the Caesar Forum was pretty awesome. The only minor annoyance was that certain flavors of Android did not like the certificate format we had initially posted on wifireg. Eventually folks figured out how to convert it using openssl and make it work.

And, not the least: effffn, mac and DEF CON would like to thank the indefatigable NOC team for their hard work. Sparky, booger, CRV, c0mmiebstrd, Dp1i, c7five, Jon2, deadication, musa, wish, johntitor, MikeD, Toph and strange do a great job and work long hours so you can internetz. Lastly, a huge thank you to Phil, Kevin, Mable and the whole Caesars IT and Encore staff for going above and beyond to make our lives easier. (and yes, that was a copy and paste from the DC 30 program)

The slides with the statistics from the closing ceremonies can be downloaded from here … Enjoy! (and yes, the mentions of the Forum Shops instead of Ceasar Forum are intentional)