Capturing flags in Thessaloniki
August 26, 2016
Earlier this summer, I have participated in the organization of a Capture the Flag competition in Aristotle University of Thessaloniki for students of the Computer Science / Informatics Department. It was co-organized by me, Spyridon Rafail Panagiotopoulos, and Kostas Mpenos, and supervised by Dr. Konstantinos Draziotis.
Organizing a Capture the Flag competition is not easy, especially in our situation where we had almost 3 days to do everything. This included coming up with challenges, designing and setting up the infrastructure, publishing information about the event, and in general doing everything that’s needed. For those familiar with such things, this is a very difficult thing to accomplish. Luckily, we managed to pull it off, with something like 4-5 hours of sleep total during these days.
Since the event was targeting students, we decided to spend the first three or so hours doing some presentations that will educate the audience and help them prepare for the main event. After all, the purpose of this was for the students to learn and not to actually find out who’s best.
That meant that we needed to spend the three days organizing the entire CTF, coming up with challenges, and preparing for our presentations. Luckily, I already gave a talk about the OWASP Top 10 in the past, so by changing a few slides and after a practice talk I was ready. In general, I don’t like to give the same talk twice, and from what I remember this was the first time I had to do it, but I think we can agree the time was extremely limited and therefore my decision was justified.
Unfortunately, the same could not be said for Spyridon, who never presented before and this was going to be his first time. He worked together with Kostas to talk about MIPS and x86 Assembly, which was needed for three of the challenges. Kostas already had some slides, but after a unanimous decision, he agreed to make new slides.
With the talks done, we then focused on the main part, which was the 4-hour CTF. Since I was the only one with past experience (and that in participation, not organization), I decided to opt-in for a “Jeopardy” CTF where contestants had to complete some challenges and receive a special flag, a piece of text, which they needed to enter into the CTF Platform and receive points.
For the platform, we chose Facebook CTF since it is brand new, easy to set up, and supported our type of CTF. I immediately set up an instance in one of my servers, which had a Quad-Core Xeon, with 8 GB of RAM, and a large enough SSD. This was theoretically more than its minimum system requirements and ran without issues. Now some of you may point out that FBCTF comes with some challenges already, so we could skip that difficult and time consuming part, but this was an entry level event so we decided to create new challenges that would closely match the skill level and the material covered in the presentations.
With the platform set up, Kostas started creating the low-level challenges that involved Assembly and Buffer Overflows. I worked in the rest of the challenges, which involved simple CAPTCHA bypasses, cookie manipulation, weak HMAC bypasses, code deobfuscation, client-side checks, Cross Site Scripting, dictionary and brute force attacks on ZIP files, Reverse Engineering, Cryptography, and other various simple things. Our goal was to make a challenge that is easy to solve and then have more challenges with increased difficulty.
The low-level challenges were written in C, obviously, and the web applications were written in Python, using the Flask micro framework. Other challenges like Reverse Engineering were written in Python as well, and contestants were given the source code.
Everything that had to run server-side ran in docker containers in my laptop, while for the low-level challenges we provisioned one Virtual Machine per team, running on Spyridon’s and Kostas’ laptops. We also set up a website with instructions and additional material for each challenge in order to help the contestants navigate in FBCTF as well as solve the challenges.
Since we did not have time to set up our own network, we hoped the WiFi available in the venue will work and we will have no problems using it. This is something that I wouldn’t do under any other circumstances and insist on setting up a new network, and for that reason we located the nearest retail shop and were ready to go and buy Access Points and switches, in case they were needed.
The day of the event came, and the three of us took a taxi to the location: the AUTh Computer Science Building in Kalamaria, Thessaloniki. This place was primarily used for offices and from what I could tell there were no classes being taught there. Dr. Kostantinos was expecting us there and showed us to an amphitheater where the event would take place. A coffee machine, cups, water, and other beverages were in the room by the time we arrived.
We were there almost one and a half hour earlier and immediately started setting up our infrastructure. I opened my laptop, checked for WiFi Networks, and luckily I found eduroam, a “secure, world-wide roaming access service developed for the international research and education community”. That meant that we could all connect to a common network since we were all students at that time. I tested for the presence of firewalls and I was happy to learn that every IP Address could reach our laptops on any port, without restrictions.
In the remaining time we completed our CTF setup, tested our laptops in the projector to make sure everything was working, and waited for our fourth speaker, Kostantinos Dimkas, who gave an interesting talk about Tor.
In the picture above, you can see the diversity in equipment used for the four presentations. Pictured is Kostantinos Dimkas, getting ready for his talk.
In the meantime, students have already started to arrive and the room started to fill. It turns out that somehow, more students, including some from ATEI Thessalonikis learned about the event and came to attend. This was very good news because we saw that people liked it and were interested in coming.
The presentations started at about the scheduled time and lasted approximately three hours. There were several questions and the audience seemed keen to learn. We were happy to see that what we worked so hard to accomplish paid off. And the fun part was yet to come.
We had a break scheduled before the CTF for lunch and to allow people who came only for the talks to leave, without getting weird looks by their peers ;-)
After the break, we grouped the participants into teams of 3-4 people and I mentioned some CTF basics, explained how FBCTF worked, and went over the rules one time. By that time, we were ready to start. I opened up the registration and the team “leaders” created an account for their team. After that, I set the countdown to four hours, and clicked the Start button.
The event started with 5 challenges, all from different categories, that were pretty basic and introductory. For the first thirty minutes or so, no team scored any points. And then, we saw the first completed challenge. This was good news, because it was then followed by more successful answers. We started enabling more and more challenges and tried to keep about 5 unsolved at any given time. This worked well for us and the participants were working all the time on solving them.
After about one hour into the CTF, Spyridon and Kostas went for a break, since we, as organizers, didn’t do much and had some free time. At this point, I started receiving a few complaints about the FBCTF page being extremely slow, and after refreshing it in my browser, I saw it took tens of seconds to load. Something was wrong. The software was supposed to be able to handle much more traffic than what the participating teams could normally generate.
I immediately ssh’d into the FBCTF server and ran htop. The system load was around 16. All cores were working at 100% but the RAM wasn’t over 4-5 GBs. I ran iftop, the htop for network interfaces, and saw traffic to the server that peaked at 120 Mb/s and averaged 70 Mb/s. This was certainly not normal. I looked at the source and they were all AUTh IP Addresses, some from the room we were in, some not.
My first reaction was to block tcp/80 and tcp/443 for every IP Address not in the building’s eduroam and AUTh networks. This was acceptable since all the participants were connected on the WiFi and were in the room. Immediately after that, the HTTP requests returned to normal, and the traffic to the server dropped to normal levels a few minutes afterwards. After restarting hhvm, everything was back to normal, and the competition continued.
A little bit over an hour later, I observed the same things as before: a high number of HTTP requests, high server load, and the like. That meant that whatever was causing this started again. This time I looked at the NGiNX and HHVM log files and discovered the paths that required a user to be logged in to get an HTTP 200 Status Code. The IP Addresses that made those requests matched the number of contestant laptops exactly. That meant I can further restrict incoming traffic to tcp/80 and tcp/443 by only these addresses and the organizer laptops.
While this may have limited the amount of traffic again, I knew NGiNX could handle way more requests per second. So the problem was somewhere else. That meant I could make some modifications to have less requests being proxied, and more answered by NGiNX.
I took a look at the paths visited and figured out some that provided public information (despite using cookies) like the currently available challenges, organizer announcements, etc. Although they needed cookies to be accessed, the response was pretty much the same to everyone. That meant I could cache these responses for 1,2,5, or any other amount of minutes and save a lot of processing in the backend. This would reduce the requests from a few hundred per minute to one per minute. Theoretically.
Before going any further, I want to point out that caching information that is only accessible via cookies should never be done. Theoretically speaking, anyone who could reach NGiNX could ask for the page and they would be able to retrieve it without running any application logic. This was done as a countermeasure, and only after the only addresses that could access the server were the participant laptops.
Soon enough, the load dropped to normal levels. That meant the system was working and the caching did not break anything. However, as we were approaching the end of the event, more team members used their laptops in order to multitask. That meant I had to allow again the entire WiFi range since I did not know the addresses of the new devices.
In the graph above, you can see the HTTP Requests / Second of the NGiNX web server during the event. They start at around 10:30, the time we arrived at the venue, and then stay low until the time the event started. The small variations in the time before the event are from the audience that visited the website since it was written in the whiteboard from the beginning.
During the CTF you can see an average of ~ 700 R/S, which I consider a lot, but apparently FBCTF talks too much. The two spikes, one at 1,300 R/S and the other at 1,900 R/S are the incidents described above. The numbers, while relatively small, seemed to max out the FBCTF server. This is something I certainly did not expect, but I’ll definitely look into in the future.
At 20:00 local time, the competition ended, and the leaderboard was finalized. The participants managed to solve 18 challenges in total, three of which only by one team. A remaining of four challenges that were enabled remained unsolved.
Although it’s not important, I am including the scoreboard for the shake of completeness.
The important thing that I will remember from this event is that we helped a lot of people learn, compete, cooperate, have fun, and of course, learn to learn.
After we packed up and started leaving, all I was hearing was people discussing the challenges, new things they learned, ideas for their projects and assignments, technical discussions, as well as congratulations given to all teams. There was no hate, no secrets, nothing. In the end, everyone was part of a single team. A team that just achieved something far more important than the prize of any CTF. And this is why I think the picture above is not important at all. Maybe I’ve added this for the funny team names. Who knows…
And as far as the future in concerned, I would happily organize a similar event any time. As a matter of fact, I am already planning the next one. Organizing these events can be difficult, time consuming, sleep depriving, but most importantly it’s fun and educating.
The good thing is that now I also have some experience. Not just technical (I’ll definitely make some changes in the FBCTF setup), but also social, organizational, and educational. I think I can make a few changes to make this even better.
So if you’re a University, especially in Greece, and are interested in hosting one of these events, please don’t hesitate to contact me and I will do my best to help you. If you’re a group of people, maybe a hackerspace, and want to host this event, please let me know as well. I’d love to work out something together.
Of course, this post must finish with a big thank you to all the students that came to the event, Spyridon for the talk, challenge beta-testing, and help, Kostantinos for the talk on Tor, Kostas for the low-level challenges, the beta-testing, the talk, and most importantly the place to stay and the food to eat during the long hours organizing this, and of course Dr. Draziotis and AUTh for having us and providing all the required infrastructure to make this possible.