Cleared for take-off: exploring the research potential of Microsoft Flight Simulator 2020

Microsoft Flight Simulator 2020 (MFS2020) launched last week, promising gamers the ability to fly anywhere on the planet through the integration of multiple technologies. The game pulls satellite images from Bing Maps and fills them with objects (such as trees and buildings) using the Microsoft Azure cloud network. The game “knows” if an object should be rendered using an artificial intelligence machine learning algorithm from Blackshark.ai.

The results are impressive and realistic visuals of anywhere on Earth, from towns and small towns to natural wonders and everything in between. The game’s ability to render objects and locations is so impressive that it made us wonder: could we use this as a tool for open source investigations? Could the game’s AI-generated 3D models be so precise to help with geolocation, and if not, could they serve some other purpose?

Over the past few days, we have been experimenting with MFS2020 and visiting cities large and small, conflict areas and even areas of environmental concern to answer these questions. We found that while the game’s 3D models are not accurate enough to be reliable for geotagging, the way the game renders topography, as well as its dynamic lighting and weather capabilities, could help in finding background by giving investigators an “impression” of the place. that they explore.

Construction of the AI ​​world (and its limits)

The aspect of the game that caught our attention first is its ability to render buildings in 3D: 1.5 billion spread across 2 million cities, according to at Microsoft. Blackshark.ai has a description of his website which explains how its artificial intelligence algorithm detects and renders buildings from satellite images, regardless of the distance from the location. Google Earth Pro, for comparison, is a must-have tool for geolocation because in addition to providing users with high-resolution satellite images, it also renders topography. If MFS2020 could add to that by providing accurate building models from anywhere on the planet, then it could be useful for geolocation tasks.

After a few experimental flights, however, it became clear that while AI has done an impressive job of rendering cityscapes in general, it has significant limitations that make it unreliable as a tool for such a precise task as geolocation.

For example, sometimes the AI ​​doesn’t recognize structures and just can’t render them at all (or puts them in the wrong places), like in the example below. district in Seoul, South Korea:

Even when the game recognizes a building and renders it, the result is sometimes inaccurate. For example, this location in the city of Maiquetia in Venezuela contains a number of buildings that were rendered at incorrect heights, three of which are highlighted below (for another example of incorrect height rendering, see this article of PC Gamer):

This is also evident with unique structures like El Helicoide, a building in Caracas that was envisioned as a state-of-the-art shopping center in the 1950s, but is now used as a prison for political prisoners. The in-game AI recognizes that there is a building there and attempts to render it, but only manages to correctly identify part of the structure:

Topography, lighting and weather effects

From what we’ve seen so far this week, the game handles topography very well, even beating Google Earth Pro (in our opinion) in a test comparison. This is because MFS2020 offers higher landscape fidelity than Google Earth Pro, including dynamic lighting, shadows and atmospheric effects, creating more realistic views. (Note: Google Maps incorporates some of these features to create more realistic views of the world than Google Earth Pro, including 3D buildings from many places around the world. We tend to use Google Earth Pro for tasks such as geotagging more often, however, because the camera is easier to navigate and position, and because it offers more tools that are useful for our purposes, including the ability to import / export KML files).

The images below were captured at the same location in western Yemen, which was the site of a Saudi-led Coalition airstrike in August 2015. The top image is the location as it appears in Google Earth Pro, while the bottom image is from the same spot and viewpoint as it appears in MFS2020:

Similar results are evident in this view, captured in Lake Louise, Alberta:

Below you can see a three-shot comparison of Mt. Ararat on the Turkish side. The top image is the mountain as it appears in Google Earth Pro; the middle image is a photograph of the mountain, while the bottom image was captured in MFS2020.

The game also includes a dynamic weather and time system that allows players to change these settings while they are in the game. Here is an example video of this feature captured in the game near the Alpamayo mountain in Peru:

The worst aspects of humanity, simulated

Just as the game can render beautiful natural sights and captivating city skylines, it can also simulate locations associated with our worst aspects as a species. The virtual world of MFS2020 is filled with the models of conflict zones and ecological disasters that we have created.

The image below shows the U.S. Customs and Border Protection Central Processing Facility, a detention center in McAllen, Texas, where the U.S. government detains undocumented migrants. The installation did securities in 2018, when images surfaced in the media showing the deplorable conditions to which its detainees – especially children – were subjected.

Use this database of the Xinjiang re-education camps of the Australian Strategic Police Institute, we found the game version of a camp in the city of Kashgar (note the most recent picture from Google Maps showing more buildings on the site):

In Ukraine, we found evidence of an ongoing armed conflict in the form of craters near Spur-Mohyla:

The game also made some buildings near the Kuzminsky firing range, which is a Russian military base used as a staging area for troops (note the most recent picture from Google Maps showing the base):

The Bing Maps satellite image of Donetsk International Airport shows only slight damage to the terminal (it is, in fact, completely destroyed to date). For this reason, the game makes the airport fully functional and even fills it with aircraft at the terminal, cars in the parking lot, and ground equipment vehicles on the apron:

In this northeastern neighborhood of Damascus, you can see that the satellite image layer shows rubble from buildings that were bombed. Wherever the game detected buildings still standing, it rendered them:

The game also makes the trees. Here is a section of the deforested Amazon jungle in Brazil, near the border with Bolivia:

And, in Venezuela, the game rendering of the Las Claritas mine offers a shocking look at the ecological damage done to this area:

Future research

Other avenues of research worth exploring with MFS2020 are its meteorological and real-time air traffic features. It would be interesting to explore how the game makes big thunderstorms or hurricanes, for example.

Since there are places in the world that appear pixelated in Bing Maps, we checked a few to see if they also appeared pixelated in-game (for a list of pixelated / unclear places in mapping services, see this Wikipedia article). We found that the game was still returning some buildings in the two locations we checked, but it would be interesting to explore other sites to understand how the game decides what to return and where.

In the images below, you can see that while the game rendered some of the buildings inside this military base in Lorient, France, it didn’t render many of those that are visible in this non-pixelated satellite image. :

The game also rendered the cooling towers and other buildings inside the Cattenom nuclear power plant in France, although the satellite image that appears in Bing Maps is both pixelated and obscured by clouds. of steam:

Conclusion

The simulated world of MFS2020 is visually stunning and immersive, making it a memorable gaming experience. As a tool for open source surveys, the game has some potential. The game’s ability to populate towns and villages with 3D models is a big part of what makes it fun to play, but as it stands, it has very limited potential to help with geolocation considering inaccuracies that we observed. More generally, the rendering of the topographical features of the game – combined with its dynamic function of time and time – could help researchers visualize the places they are working with.

Simulating a 1: 1 replica of our planet with high graphical fidelity and populating it (if imperfectly) with objects is an impressive feat and a clear springboard for future simulation technologies.

Thanks to Aric Toler for his help with this article.