East of the Rockies: Augmented Reality to Tell History
1944. Following the attack on Pearl Harbor, all Japanese living in Canada are considered as enemies of the state. Under the orders of Mackenzie King, 22,000 Japanese Canadians—including the young Joy Kogawa and her family—are taken from their homes and communities and interned in camps in the Rockies.
Joy Kogawa is an author, a prominent figure of Canada’s literary hall of fame and a member of the Order of Canada. Today in her eighties, she channelled this dark chapter of her childhood and Canadian history into a powerful and authentic mission in defence of human rights and social justice that characterizes each and every one of her works. It is when he read her novel titled Obasan, dealing with Joy’s years of internment, that Jason Legge, the project’s co-producer and co-creator, felt compelled to do something. Already a great admirer of the author’s vivacious spirit and unique perspective, he wanted to give her the opportunity to take part in the development of an interactive project.
As the country was preparing to celebrate its 150th anniversary, it appeared fully legitimate to him to take advantage of the opportunity to share a rather unknown facet of our history characterized by resilience and tremendous courage. The producer consulted Google and learned that the author lived only a few blocks away from his studio. So he decided to contact her and let her know that he was interested by her story.
“She had never played a video game before! So we invited her over and played PlayStation with her for an entire afternoon. We selected games that were interesting from a narrative perspective to show her what we wanted to do with the story,” explains Jason Legge.
A project not quite like any other
Augmented reality proved to be the technology that would enable players to immerse themselves in this universe situated somewhere between literature and interactive storytelling. Amélie Rosser, Jam3’s senior developer, was a member of the team from the onset of the project. And they hit a real home run when it came to the casting. “Code has always been an art for me. I’m very artistic and creative and that’s what I love about what I do: to produce interactive art. I honestly think that AR is the most exciting thing that happened lately technology-wise. It’s so much more fun to see something in your physical space, to see the game emerge from it! It’s a game changer!”
And that’s without mentioning that it was a touching subject for all of the team’s members the moment they first met Joy. “I felt entitled to the author. She faced discrimination as a child and it became such a personal thing for me. I wanted that game to do it justice and show people how to be nice to each other. I felt a strong emotional connection and when you share your beliefs with a project, you really feel more connected to the final result,” says Rosser.
It is this very strong emotional bond and this enthusiasm that enabled the team to come together and overcome the obstacles along the way. It must be said that tremendous challenges awaited them…
The team got together and became stronger as it embarked on what would be two years of research and development, uncertainties and immense pride. The project was first carried by its two creators, Jason Legge and Dirk Van Ginkel. Jam3 and the NFB, one of the studio’s long-standing partners that was taking part in an original collaboration for the very first time, joined on later.
Creative chaos and technological uncertainty
Funding, creative process, technology, storytelling, design, management: there was uncertainty from beginning to end. On the one hand, the developers learned to use AR with Unity (a highly popular multiplatform game engine). On the other hand, Jam3 was developing its very first game and the project’s artistic direction raised major questions. “We had no idea how we would do this!,” candidly recalls Rosser.
In both storytelling and creative terms, it was essential to first come up with a flexible modus operandi, a system that would enable everyone to work together and advance despite frequent changes of direction, the distance and obstacles. In other words, there needed to be structure in the chaos.
However, before the project could begin, they needed a script based on Kogawa’s novel that could then be translated into an interactive story. The team entrusted this part with Walt Williams, who came back with an 84-page bible that would become the heart of the game. The story would be interpreted by Yuki, a young 17-year-old who was interned in a camp in the Rockies along with her family. In her diary, she documents the happier and sadder days of her new life. To add meaning to the experience, Yuki’s voice will be interpreted by none other than Joy Kogawa’s granddaughter.
“The script was structured in steps such as to create a linear action. These three acts were divided into interactions, the equivalent of a scene-to-scene, and then the storyboard led to the game itself!,” explains interactive designer Steven Mengin.
Augmented reality, the interactions, the game play mechanics, lighting and visual effects monopolized the discussions during the initial technical meetings. And don’t think for a second that Joy Kogawa was overwhelmed by all of these technological discussions!
The first act, the work’s prototype at its development stage, proved to be quite a challenge—each success having resulted from a cascade of trials and errors. “The learning curve was pretty steep!,” admits Rosser, laughing. And that’s not to mention that the use of AR engine Vuforia gave migraines to the entire team until Unity proposed AR Foundation and practically saved the day as well as the mental health of the entire development team! Six months down the road, the team had accumulated an immense pool of know-how, yet it didn’t have much to present, especially not in terms of a testable functional experience. It was a hard blow for a team that wanted to maintain the relationship of trust between the partners but that could unfortunately not skip any of the steps. All in all, everyone knew that success would eventually follow. They just didn’t yet know how it would…
When a true story comes back to life
The storyboard completed in one month based on the script made it possible to structure the scenes into interactions and then to separate the interactions into four categories based on their level of importance for advancing the game’s script, from “essential” to “rewarding” on a second level for the storytelling.
Each storyboard table was then assigned a coloured disc (as illustrated in the visuals below) to place it in the linearity of the user experience. The types of interactions were then defined (point, scroll, align the points, get closer) and incorporated into the storyboard. The process here was iterative, based on trial and error and the team had to work on it (practically) up until the last minute.
It was quickly confirmed that the characters would be recorded via motion capture to give the team the most freedom possible. This decision proved to be a rather major error, especially for the development team, which was responsible for cleaning up all of the images to render them useable. Where they believed that had saved time, they lost in terms of ease of movement, flexibility and character customization on screen. It’s impossible to always hit home runs!
What is the dilemma facing any team that works in interactive? Proposing the most sophisticated experience possible in terms of design and interactions and offering the audience an accessible and convincing experience. This industry reality proved to be one of the project’s most heartbreaking aspects. However, far from fighting over it, it’s by working together that the teams made the best compromises.
“Designers always want to make all these crazy effects and we didn’t want to turn down suggestions because it’s such a bummer for creatives when we always say no to their ideas! So we’d always work to come up with an alternative. When people share their ideas for a project, it’s in your best interest to respect it and find the best approach for both sides.” – Amélie Rosser
“We worked together all along the way. Often, the designers would sit down directly with the developers to add noise and colour, move an object around, adjust the size of the text or the quality of a button… You’d never think that it would take that long to obtain such quality of animation and fluidity, but it’s incredibly long! The devil is in the details.” – Steven Mengin
If there is one thing that the team of developers learned during the process, it’s that it’s fundamental to have an architecture and a nomenclature that are sleek, standardized and shared by all of the members of the team. And that’s in addition to documenting the libraries, the animation mechanisms, the textures, the UX and so forth.
“We simply cannot afford making mistakes when we have so much to manage! We needed a master, a source of ‘absolute truth’ that conveyed the project perfectly to everyone. It was very complicated to make the adjustments, to work with Google Slides, to convey all of our ideas with respect to development, artistic direction, UX, etc.” – Steven Mengin
An environment faithful to the lives of the internees
The textures and lighting effects that change as the days and seasons unfold are at the heart of the game’s artistic direction. From natural light to the light of the oven representing the sole source of interior heating for the interns, light contributes greatly to this poetry. It’s a cozy atmosphere in which combinations of shadows and beams of light are at play.
To create the village’s environments, the team worked with Aaron Campbell, a Vancouver artist who uses Photoshop. He created authentic locations and scenes somewhere between dream and reality. Moreover, the team met with former camp interns to validate the veracity of everything from the locations, field crops, objects in the houses and even the dimensions of the rooms. In parallel, a team of photographers made its way to British Columbia to document the before and after as well as to provide the team with visual cues.
“Incorporating augmented reality in the artistic direction was difficult, but we found the right solution by adding areas of fog along the coasts to have a real-world scene emerge from it. Another sizeable challenge? It’s called East of the Rockies, but where are the Rockies in the game? In terms of perspective, how can the Rockies be incorporated into it? It’s impossible to create a backdrop of mountains when all of the scenes are focussed on the story’s characters. In terms of proportions, it was not at all realistic. We therefore decided to use the mountain as a metaphor.” – Steven Mengin
They ended up being incorporated into several menu items as well as into the project’s main visuals. The result is a smooth experience that is respectful of history.
A popular and critical success
For the project to gain in popularity as hoped by its creators, it was essential that the game be available in the App Store. The final sprint to plan the game’s launch and going back and forth with the Californian giant ensured that the game made it to the prestigious rank of App of the Day on March 15. It’s a home run in terms of visibility that led to close to 100,000 downloads in the first three weeks.
Each time the team receives photos of youth who are interacting with the app, they are filled with a level of pride that justifies all of the overtime hours invested in the project, not to mention all of the pizzas ordered along the way.
For Legge, this project revealed an obstacle that remains all too present when it comes to interactive: representativeness. “We have to admit that we have a representation issue in interactive storytelling. Working on a multigenerational story with an 84-year-old Japanese Canadian woman was so incredible. I can humbly say now that we’ve become friends. I think that we’re able to reach a new generation with this story and the medium: it’s a challenge that we overcame.”
With East of the Rockies, Jam3 rose to a seemingly insurmountable interactive challenge by proposing a successful experience. It’s an immensely sensitive project that sheds light on a chapter of our history that remained in the shadows for too long.
