(This picture is actually from the closing ceremonies in 2008 in Beijing- the Rio Games are barely a day old, so the internet isn't exactly awash with images to snag yet- but you get the basic idea.)
The main symbol on the flag is obviously, the Olympic Rings, which were designed in 1912 by Pierre de Coubertin, co-founder of the modern Olympiad. At the time, the rings were meant to represent the five world continents- Africa, Asia, America, Europe and Oceania- while the colors of the flag- the five rings and the white background were meant to represent all the colors that appeared on all the national flags of the countries participating at the time.
The bigger the Olympics became, the broader the definition of the meaning of the flag became as well. Per the (very extensive) Wiki-Page on the many, many Olympic symbols out there, the current view is that the symbol stands for an international movement that welcomes all countries to join- with the five rings representing the five continents of the world, interlocking when the athletes meet at the Olympic Games.
Interestingly, there's a little bit of a contradiction when it comes to the whole, 'which ring stands for which continent' debate. Prior to 1951, blue was for Europe, yellow for Asia, black for Africa, green for Australia and Oceania and red for the Americas- but that was removed from the official definition of the symbols because there was no evidence to suggest that de Coubertin intended the rings to represent a continent based on color. Despite that though, the Association of National Olympic Committees places the logo of its continental association inside the rings of the corresponding color.
So, I'm not sure what to make of that. Seems like a weirdly unofficial/official definition of the rings/continents thing.
If you think that's confusing, well, then let's dig into what the hell happens to the Olympic Flag between Olympics. Turns out that flags hoisted over a host city stadium for the duration of the games get retired at the end of the games- but, the flags that are used at the closing ceremonies are passed on to the mayor of the next Olympic City in what's called the Antwerp Ceremony.
TL;DR of this whole mess: basically, the first flag was presented to the IOC in 1920 at the Antwerp Games. Afterward, they couldn't find it*, so they had to make a new flag for the Paris Games in 1924- that flag was passed to the next organizing city of the summer and winter games until the Winter Games in Oslo, when they made a flag just for the Winter Olympics to pass on to the next Winter Games while the flag from 1924 continued to be passed along until 1988, when it was retired in Seoul. (And thus, instead of the Antwerp Flag, it's now the Seoul Flag that gets passed along between Summer Olympics.)
I'm not even going to get into the Olympic Hymn. You can read the Wiki-Page for that whole mess. In the meantime, until next time, keep your flags flying- FREAK or otherwise!
*They did actually end up finding it... in 1997- and got it officially returned in time for the Sydney Games in 2000.