XR in the enterprise

Podcast transcript

Alexey Boas:

Hello, and welcome to the Thoughtworks technology podcast. My name is Alexey Villas Boas. I'm the head of technology for Thoughtworks, Brazil. And I will be one of your hosts this time, together with Rebecca Parsons. Hello Rebecca.

Rebecca Parsons:

Thanks Alexey. Hello everybody. My name is Rebecca Parsons. I'm the chief technology officer of Thoughtworks, and excited today to talk with our guests, Margaret. So Margaret, would you please introduce yourself?

Margaret Plumley:

Thank you for inviting me to be here. My name is Margaret Plumley. I'm a product strategist in the XR practice lead for North America. Very excited to be a part of this podcast.

Alexey Boas:

Happy to have you with us, Margaret. Thank you so much for joining. And so the conversation today is about XR, extended reality. And then why don't we start with the definition? So Margaret, can you guide us through what extended reality is, and the difference between all the acronyms AR, MR, VR and et cetera?

Margaret Plumley:

Sure, absolutely. So, AR, augmented reality, MR, mixed reality, VR, virtual reality. The X is often used as a variable to mean any of the three, or extended reality. And VR generally means if you have a full headset, and the entire world is virtual, versus AR where you're augmenting the world that you are in, and mixed reality is somewhere in between the two generally, used when you're referring to smart glasses. But yeah, lots of acronyms, and you can even add more technologies in there and combine it with things like IoT, and have all the acronyms you want.

Alexey Boas:

Well, and it's definitely getting more and more interesting as time goes by, but at the end of the day, these are not new technologies, right? So, how have they evolved? And what has changed? What is new these days about these things?

Margaret Plumley:

You're right. A lot of these technologies, especially virtual reality have actually been around for quite some time. The biggest changing that we've seen, is in the hardware price and weight. The price has gone down by about a decimal point, so has the weight and the number of cables. In fact, you can for a couple $100, buy a very good virtual reality set. And also the software is off-the-shelf. Gaming engines like Unity and Unreal are being used. So you don't have to write everything from the ground up. And the consumer applications of course, are the ones that are driving this. So they're driving down the cost, and they're driving up the usability of things like Unity and Unreal, and enterprises are able to harness this. We're seeing these consumer headsets are being used in everything from training for retail, to training for medical students. And the enterprise itself has gone from proof of concept to KPI, because they are actually seeing the return in using these applications.

Alexey Boas:

Yeah, that's cool. It's really like the technology is going beyond just a cool booth at an event to real use cases inside the enterprises, right?

Margaret Plumley:

Yeah, they are. They've really gone from... Most people when they think of AR, VR, they think of games, they think of consumer, but they've actually been used in enterprises with really positive return. You're seeing actual cases where time on task is reduced by anywhere from eight to 30% in industry. So, there are some very real enterprise level applications out there with very real results.

Alexey Boas:

And from a software development, or from an application development perspective, so how do things change when they develop things for XR versus developing a web application, or other technologies that are more common?

Margaret Plumley:

Well, the world is the interface, right? Even on AR, when you're using your phone, it's not just the rectangle that you're designing for, that rectangle is just a magic window into a larger world. And this changes how you think about designing. You're no longer drawing shapes on a flat surface, you're putting objects in the real world. And it also changes how developers need to think. About 20% of it is a new scripting language, but 80% is a whole new world. You're creating 3D objects, you're applying shaders and lighting, and it's actually a 4D world, because you also have time. You can have all the triggers that you have in a gaming world, and you can apply them to the real world. So, you can use these gaming triggers in the industrial world, where you can walk up to some sort of machinery, and it shows you the IoT in your mixed reality glasses. So you can look at a piece of machinery and it shows you, what the current status of it is, but that also means that development and design, you need to think in terms, not just all three dimensions, but also in terms of these triggers that go with it. But because of that, it's also an exciting new challenge, both for the designers, and the developers. There's a lot of innovation here, there's a lot of green field, and a lot of opportunity.

Rebecca Parsons:

Well, isn't it also conceiving of what the solution to the problem even looks like? I mean, we as technologists have been trained to look at a problem, and think of a solution to it. And when you think about, okay, how would I imagine a customer support interaction within a virtual world? Even how you think about addressing the problem is a challenge. So, what are kind of the skillsets you think people really need to bring, in order to think about the world as the interface?

Margaret Plumley:

That's a good point. And it is a question of making sure that you think about the entire ecosystem, and you don't turn the technology into just a shiny thing. Anytime you have a support issue, you might not necessarily need to have a remote assist AR application, but you do need to have some sort of a support ticket. So, it really is a matter of looking at that entire ecosystem, that goes with the problem that needs to be solved, and what technology you do and don't use. AR isn't always the answer. You shouldn't need to download an app, have the latest phone, have an unlimited data plan, to be able to find the entrance to a hospital, right? Sometimes you just need a big arrow on a sign. So, it's really important to think about, what is the problem that you need to solve? And what is the right tool to do that? And how do they all interact with each other?

Margaret Plumley:

And how does it interact with the entire ecosystem of what you're doing? If you're building a training system, you want to make sure it interacts with the LMS, with the learning management system, same for any of these other things. If you're doing AR shopping, then you want to make sure that it interacts, not only with your shopping cart, but also with your stock system. You don't want to show some incredible thing that they love, but you know what? That was last season, and you don't have it anymore. So you want to think about that end to end, not just the end to end user experience, but also what is that end to end technical solution.

Alexey Boas:

Yeah, I find it very interesting, what you say about how design changes, and when we think about more traditional applications, the development of web application, or even a GUI application, you always have that frame of reference, which is the screen. And you can use that as a powerful limiting resource, to think about the interactions and the things that people do, but that changes completely when you have the world as the interface as you put it. So, I wonder how the design process itself changes with things like even prototyping. So, do we have the tooling to do that, or the techniques? So what have you seen in that sense?

Margaret Plumley:

Well, you see a little bit of everything, and trying new techniques is always part of the fun of designing something new. And it's also important to understand where your old techniques don't work. If you immediately start with a 24 inch screen to design an application, even for mobile, it's not going to work. So if you are designing for the world, you want to prototype in the world. I mean, I've sat around when I'm thinking about doing things, I'm sitting in my house, and I'm putting tape on the floor, and I'm hemming things up on the wall and I'm saying, "Okay, what would it be like if I were to walk through, and I was looking for this? What would I want to see? And where would I want to see it?" So, it's actually a lot of fun. You get to be very creative here, and you get to go back to cutting things out of paper, and using tape and scissors to put it up, but it actually works. You need to build these sets, and I think, anybody who has a background more in theater or in architecture, because they're used to moving through a world that becomes very helpful when you're designing for this world, because you do have to design differently.

Alexey Boas:

Yeah, it's interesting you mentioned other areas, and architecture, and theatrical arts, because from one perspective, some of these techniques, they have been very developed in gaming, and in filmmaking, right?

Margaret Plumley:

Yes, they absolutely have. A lot of the techniques come from there, a lot of the tools come from there, and it actually... You have a lot of the talent comes from there as well. I mean, I used to work, as you know, in the film industry where I worked with some amazing artists, and a lot of them have gone on to be working in VR, in AR, and building these kinds of worlds where you can build, because you have that level of talent, you can build hyper-realistic worlds, that they are fully immersive, which is part of why it's so successful to do things like training in VR, is because you can recreate this entire world. I mean, you have applications where doctors learn surgery in VR, right? Because you can create that world. You can create things to help people practice these sort of rare and dangerous cases.

Margaret Plumley:

Like, how do you learn how to handle working in construction, on a freeway? Well, you kind of don't want to just throw somebody on a freeway with trucks going by, but you can create something in virtual reality. And those are one of the brilliant cases, these training applications are so fantastic. And because you do have a lot of the tech, and a lot of talent that comes from movies and games, you can create a fantastic environment in which to do these things. In fact, there've been really fantastic results, not just in terms of the ability to complete a task, but if you look in the medical world, on the one end of it, there's doctors being able to practice surgery. I mean, how do you practice surgery on a human being? Do you want to be the first time somebody has done that surgery?

Margaret Plumley:

Do you want to be the doctor doing it for first time on a person? Having the ability to do it in VR, is helpful. And also for patients, like there've actually been studies on where, before a patient goes through a procedure, if they are able to experience it in VR, then when they go through it for real, their fear is reduced. And therefore, their response to the treatment is actually higher, because they don't have that stress aspect. There are a lot of incredible results that you get from this training because you can't create a whole reality.

Rebecca Parsons:

Now, you've talked about some of these really exciting applications, but if so much of this technology, and so many of the approaches come from the gaming world, what about making it enterprise ready? What about making it something that a medical school would feel comfortable turning their doctors, or their medical students more precisely loose on? What have you experienced in terms of moving these processes and techniques into the enterprise environment?

Margaret Plumley:

They've actually been very successful. And you see that in training, you see tremendous success, you see tremendous success in medical, but a lot of times, you don't see success because it's happening in silos. And I think that's where a lot of it falls down, is if somebody tries to turn it into a shiny thing, and doesn't take into account the rest of that whole experience, then that's where you don't have the success. If you want to create something that is a really good experience, it needs to be, again, end to end experience for both the person who's experiencing it, and for the company that's building it, right? It needs to take into account, training the LMS, you need to take into account, like, what is the right answer? Right? Is this the right time to do VR? Is this the right time to do AR? What are we trying to achieve here? So, there is success when it's done right. You just have to make sure you think about, how do you tie it together? How do all of the pieces hook together? If you look at something like even task completion, you're seeing tremendous gains in that because people have the tasks lined up in front of them, but unless you build that properly, and unless you hook it up to everything, it's not going to work.

Rebecca Parsons:

And we've talked a lot about the design aspects of this. When you think about the software development life cycle as a whole, one thing that comes to mind to me is, the differences in testing. Just the number of degrees of freedom, that you have in this four-dimensional world as opposed to this, first this will happen, and then, and that, and that. What can you tell me about approaches, particularly in these more mission critical settings, for thinking about testing?

Margaret Plumley:

Yeah, that's definitely a big issue, because you need to think, and you need to rethink, like, what's your schedule for doing anything? And what is your methodology for doing everything? Because when you're testing, you'd have to make sure that, did the code do something, but it's very important that did it do it right? And can you apply your old methodologies to it? Like you want to do continuous testing, but that interaction, and there is the complexity of multiple devices. If you are in say a training in a school situation, then you can control what devices people are using. You can say, okay, you're using this model of this headset, and test just on that, and test exactly what people are doing, versus if it's a broader sense, if you're doing something where it's more... You're building something for your customers, and you don't know what device they're using, you have a much broader test or testing spectrum, and you also have to consider multiple things like, do they have the data connection?

Margaret Plumley:

Do they have the app? But then you want to make sure that you're testing the right thing like, do people just learn how to game the system? Right? Are they learning how to respond to the triggers that you built up? Especially in a training system, this is so important, that you make sure that they didn't learn how to react to your training system, like back when we were all playing Pac-Man, you learned exactly which moves to take to finish the level, is entirely different to learning how to actually complete a task. So you do have to think about your testing in a different way, and part of it is the mechanical tools, and part of it is actually, what are you testing?

Alexey Boas:

Yeah, that kinds of goes back to some of the things that you're saying, and some of the challenges for the broader adoption of, thinking in silos, and thinking about the whole system, the whole integration with the rest of the enterprise. And when we think about the enterprises, that sounds interesting to me, because it's connected to some of the other things that we usually believe in, like have, well, integrate the teams, multifunctional teams, we can think about different aspects. So you will probably need people who know about the specific AR, VR, XR technologies, but also about the business of the enterprise, and also about the rest of the technology ecosystem. So, is that something that is key for, in your opinion, for a successful adoption?

Margaret Plumley:

Oh absolutely. You still need to make sure that you, again, going back to those core questions of, what problem are we trying to solve? Who are we trying to solve it for? And what is the right answer? And making sure that you do have the right people involved. It's like any new technology. You don't want to just take the new technology, and try and stick it on there, if it isn't the right answer. So you want to make sure that you have the right skillsets, but that you're also, listening to everybody else who's involved, everybody from training to support, because anytime you add a new technology to your ecosystem, you add all the other aspects that go with it. You add the support, you add the training, you add all those other parts. So, you do need to pay attention to all of those, and make sure that everybody knows how they fit together.

Rebecca Parsons:

So, can you tell us about some other potential applications, or application areas that we haven't really talked about? And people think of things like training, and that's kind of easy to get your hand around, but what other kinds of applications that have been explored? Or thought about?

Margaret Plumley:

One of the ways that you see this technology being used, is just exploring and experimenting. If you look at places like NASA, they want to walk on Mars, right? And this is where you combine the technologies, where you take the images coming back from the rover, and you can build a 3D world, and then you have scientists around the world who can put on their magic glasses, and they can all be on Mars together. And you find that this is especially important now in the era of COVID, where people are apart. So you can sit there, and you can walk together on Mars, and you can look at things, and you can point at things. You also have other cases where you can glue all these technologies together. You think about any event, like say, you're running in a marathon, and you want to know, how far am I from the entrance to the exit?

Margaret Plumley:

Where's the nearest restroom? You can have a watch that buzzes, that shows you in one symbol where the next water station is, at the same time, the friends who are watching you run, can pull up their phone, and see where exactly you are, so they can get that picture of you coming around the bend, and the people who are running the event, know where everybody is in case there's an emergency, and they need to get people out. And you can apply that to any kind of an event where you combine all of these technologies, and they have different uses for all the different user types. A lot of times when you build a solution, you're building it for one target user. But in that case, I just mentioned three. I mentioned the person who is running, I mentioned the person who's watching them, and I mentioned the people who are running the event. And you can think about tying all these technologies together to serve multiple people in it.

Margaret Plumley:

And then, another one is when you look at things like travel, and history, and learning about a place that you're in, you can have things like AR apps, where when you go to these historic locations, you can actually relive events in those places. So if you think about tourism and traveling to different places, and again, you can combine technologies where if you need to rejoin your tour, the map in your phone knows exactly where you are, where you need to be, when you need to be there, and can tell you when to get back, but meanwhile, it can give you all the historical context of what went on, where you're standing.

Rebecca Parsons:

Well, and one application I saw a couple of years ago, which I thought was fascinating was, they were actually using a virtual world as a data visualization technique. So, you would map the data into this world as objects in the world, and then you could manipulate it. And you had various gestures to zoom in, to get more detail of a data set in a particular space. And it was really quite natural watching people, just sort of walk into this part of the room, and they were sort of engulfed in this hologram like thing, and here they are manipulating a data set and trying to draw conclusions from the data, with it being presented in that way. And for inherently three-dimensional data, that's actually quite a rich way of thinking about data, as opposed to trying to figure out how I can on a screen, really get the kind of three-dimensional projections, whereas you can be in the middle of the three dimensions, and really get a sense of it. So I thought that was actually quite an interesting way of using it, and taking advantage of that inherently three-dimensional nature of a world.

Margaret Plumley:

Yeah. And you could do that of thing, and you could walk up to an actual object, and see the visualization around that object, and see how does air flow around the aircraft, see the lifts, see the drag, actually move things in real time, and see how they change. You'll see this sort of thing a lot in architecture, and construction, where you can stand on a construction site, and you can show, how the construction site will change over time, and how it will impact lines of sight, and how it will relate to the environment. So being able to walk out to say where a freeway or where a building will be built, and stand there and say, "Okay, this is where the traffic will flow." Or, "This is how high the building will be." Or, "This is where the pipes will go, when they're built." You're seeing this a lot in architecture and construction actually, as you're seeing the guys who are going to wire the building, walk into this empty space and say, "Okay, I need these wires here. I need those pipes there." And being able to see all of it as you're standing there. So it's really incredible to be able to see it on site, where it's going to be.

Alexey Boas:

And one thing I find it hard to get my head around when I look at some of these things is, the sheer number of different areas. And of course, in my case, it's likely because it's far away from the things that I know. So you have a new technologies, you have to think about design changes completely as we've discussed, testing changes completely. You have even to think about things like graphical design, textures, sound. I know this might be a tough question but, do you have any advice for an enterprise software developer trying to learn some of these things? So, where to start, just take a look at Unity, or have a look at the tooling, or are there other things that people who want to learn more about how to develop these kinds of applications should pay attention to?

Margaret Plumley:

Yeah. There are a lot of professional organizations out there that can be a tremendous resource, both in terms of just inspiration for what people are doing in this area, and also in terms of information for how to do things, and how other people solve problems. I mean, there are newer organizations like the VR, AR association, which focuses obviously very clearly on that. There are other organizations, like for instance, when you look at SIGGRAPH, which is computer graphics, which in the past has been a lot of scientific visualization, and movie-making, and gaming, and now has a lot of AR, VR as well. So there are tremendous communities around this. There're also meetups, and a lot of tutorials online for learning how to use things like Unity and Unreal. So there are a lot of professional organizations out there to look into, and just see what people are doing, and how they're doing it, and get inspiration from them. And I really recommend going to a lot of these conferences, and just keeping in touch with the industry, and what's going on.

Rebecca Parsons:

Great. Well, it seems like we have an exciting future ahead of us.

Margaret Plumley:

Yes we do. It's really an exciting technology, an exciting green field, and it can be anything you want. You can build other worlds, or you can augment this world. So it's really exciting to see how it's coming together, and how people are pulling multiple technologies together, just to make a more exciting place.

Rebecca Parsons:

And multiple disciplines. I mean, I was involved with some people who were putting together a university college around digital arts, and they had screenwriters, and choreographers, and visual artists, and composers, as well as technologists, and all of the new kinds of collaborations we're going to have to work out. That should prove interesting for our industry, but I think it's really exciting.

Margaret Plumley:

It is. And if you look at places like NASA, they have people who used to work in the film industry, they are helping them to imagine these worlds that exists. How do you explore space? Well, it's a combination of a scientist with the facts, and an artist with the imagination, to try and communicate it back to your common person. What does Jupiter look like? What does it look like to go to an exoplanet? What is this experience? So there is this fantastic combination of art, and science, and technology that's coming together.

Alexey Boas:

And we're coming to the end of episode then. It was a great conversation, and a lot of joy to have you with us,

Margaret. Thank you so much.

Rebecca Parsons:

Thank you Margaret.

Margaret Plumley:

Thank you for having me here. It's always good to talk about new and exciting technologies.