Hi everyone - I'm excited to share insight into the Player Portal History that was added back in February.
This entry of the Dev Diary was provided by @ofer2 , Senior Software Engineer.
===========================
Hi everyone, my name is Ofer and I was the Feature Lead for Portal History. With this Dev Diary entry, I wanted to give you all a bit of insight into the development process for this feature.
Competition, exploration, and social are three major aspects to Ingress. When I worked on Battle Beacons, I got into the competitive aspect. But with COVID and staying at home, this was put on the backburner. After being stuck indoors for so long, I was really looking to lean into the exploration aspect of the game. The main problem here, for me, was that I love exploration, but I need a goal and I need to see progress towards completing that goal. Therefore, like many of you, I wanted to be able to see the portals I had been to and go to new ones until I CAPTURED THEM ALL!
The opportunity to dig into this arose last year when there was a call for ideas for a team-wide hackathon. Our pitch was to add code to the server to track all the Portals you’ve captured and show it to you on the app’s Map Screen. In this proof of concept, this wouldn’t include previously-captured Portals because that was a whole other can of worms, which would definitely not be doable within the scope of a hackathon.
The main challenge with this limited feature was that we needed to mix spatial data with Agent-specific data. Normally, when you make a query for entities on the map, all of that data is the same for everyone. Therefore, the code responsible for doing that query has no idea which Agent is the one that is asking for the data. Secondly, we needed a way to ask for that Agent’s specific data in an area without having to look through all the data.
The basic plan was to create an entity (a storage container for data) that was keyed by player ID and S2 cell ID. An S2 cell is a way of dividing the world into chunks and labeling it. You can read more about it here if you are interested: https://s2geometry.io/devguide/s2cell_hierarchy.html. Then, when you query for entities on the map, we would route that through a new interface which also looks up the player data using the S2 cell and “mixes” it in to the Portal information on the way out.
With this plan in mind, I recruited Andrew, another engineer on the server team, who was interested in the feature and set off to work. At the end of the hackathon, we all presented our projects and I had the honor of presenting my team’s:
Showing unique Portal captures on Ingress Prime: Image by ofer2
We used red dots at the bases of Portals and only showed unique Portal captures here, but programmer art aside, I think we made a really awesome feature, and wanted to work towards building a real version for production.
After the hackathon concluded, we returned to our normally scheduled work, but the team saw the results and the opportunity to bring to life a feature that Agents have been asking for for years. Brian, however, wanted the whole enchilada: prior Portal History included together with all Agent history from the beginning of the game AND to top it all off, he wanted it on the Intel Map, too.
Brian tasked the Engineering team with getting this feature out. Fortunately, we had a strong foundation with our hackathon project. Alas, it turned out to have a few kinks: returning Agent-specific data for every Portal is often unnecessary as the data hasn't changed since the last time you interacted with the Portal. To tackle this, we return the player data as a completely different piece of data from Portal data, and then put it together on the client. Unfortunately, the Intel Map doesn’t have real-time updates like the client does (it refreshes all data), so it still had to have the player data attached to the Portal.
The second major part of this was pre-populating the data with the entire history of Ingress gameplay. When we initially looked into this, we saw that the records storing actions that Agents have taken have a field for the entity. This field is hashed, meaning that we would have to reverse the hash to find out the actual entity that the record corresponds to. In order to do this, we created a process that would go through every single Portal ID and hash it so that we could use that hash to compare against the Agents’. Then we would go through each Agent’s records and find all the Portals they’ve visited and captured, and write that data to the structures that the client will query. After we finished writing this, we ran it in the dev environment and it wrote out the results to the datastore.
When we checked our results, they didn’t match the Portals we expected. It turns out that we made a mistake when examining the code that wrote out the tracking records. The function takes in a string which it calls ID, hashes it, and writes it to that field. The thing we missed was that this function wasn’t called in the case of visiting and capturing Portals. We instead called a different function first, which combined the ID of the Agent with the ID of the Portal. This means that the single list of hashes that we made before wouldn't work. We had to recreate the list from scratch for each individual Agent.
The cost for doing that process once was high, though reasonable; but multiply that for every Agent and the end-to-end process wasn’t feasible. The team put our heads down and went to work trying to reduce the cost. After some diligent effort and time optimizing the pipeline, we succeeded in reducing the cost significantly. One of the optimizations that helped us to do this was by making the flow run on a single machine per player instead of spreading it out to multiple machines. This involved some creative ways of reducing the size of the data that each machine would need to have in order to generate the results. Then, by specifically choosing to run the optimized backfill for Agents Level 3 and higher, we managed to get the cost into a reasonable range. The cost reduction of the engineering team here allowed us to give this feature to all Level 3 Agents and higher without making it a paid feature.
At this point, we had a working client and server, but we needed a clear way of displaying all of this data. We came up with a clever concept of “Layers.” This enabled us to only show Agents the information that they care about. The Art team took this on and came up with an initial concept, which everyone was so excited to try out. The first iteration was hard to read, so we continued to test different designs. The Art team quickly took that feedback from the team and from Vanguards, and increased the size of each Portal History ring and added spacing to the Portal, which made the rings easier to see at different zoom levels.
We have iterated quite a bit on this feature from the hackathon to during development before the launch all the way to development after launch as well. There were some additions that we knew players would want (e.g. invert on the intel map) but even without that, we knew that players would really enjoy this feature, so we elected to ship the core feature and then continue to iterate after launch. This iterative design process enables us to draw feedback from Agents and our teammates, and help guide the direction of the Ingress roadmap.
Player portal history was always something that we wanted to put on the roadmap as it was one of the top requested features from players, so we found a creative way to get it out to players. We’ve since expanded the Layers button to include Drones so Agents can see more activity going on around them. We hope to continue to make further enhancements in the future!
