Irena Cronin, CEO of Infinite Retina, interviews Victor Luo, General Manager, Operations Lab, and Sasha Samochina, Immersive Visualization Producer, at NASA Jet Propulsion Laboratory (NASA JPL) on NASA’s Virtual and Augmented Reality initiatives.
A revolution is happening in Hollywood. Decades ago movies were made with physical sets, cameras and lights run by people, with actors interacting with physical objects, er, props.
That Hollywood is quickly disappearing and here we visit the largest volumetric movie studio in the world to see just how.
The dome we visited can fit almost an entire basketball game inside with 100 high resolution video cameras aimed at the center of it, along with a datacenter of computers that hold the captured images inside, and let a new style of director build movies and TV shows in a whole new way for new devices that are just coming on the market now (virtual reality, 1k per eye today, with higher resolutions coming over the next few years).
At Infinite Retina we are studying how industry is changing, one of the industries that is already seeing massive changes due to Spatial Computing technologies (computing you, a virtual being, or a robot, can move through, like AR or VR) and Dave Smiddy, Head of Product at Intel Studios, details some of the changes that have already come to the movie/TV industry and how he, and Intel, which has put millions of dollars into this studio, sees the changes coming.
“We have seen growing numbers of digital artists and their jobs growing,” he says, showing where the job shift is underway. “All the backdrops are CGI,” detailing why the dome is green in color all the way around.
Costumes are digitized. “We are putting mocap suits on people.”
“The golden age of TV is back,” he says, while pointing out that these kinds of techniques let directors build rich stories with lower budgets thanks to this kind of technology. But then he talks about how people in the future might want to walk around movies and interact with the actors, who might not be actors at all. At Tribeca right now, for instance, Fable Studios is showing off Lucy, which is an animated entertainment experience where viewers can interact with the main character and walk around her world, and even pick up objects and play with them, sort of a hybrid of a video game and a movie.
“The areas which it gets interesting is where we combine AI with actors to create the end materials,” as we discuss the development of virtual beings like “Mica” that Magic Leap, which just saw another $280 million invested in it last week.
“It will be generated by massive amounts of compute,” he says, which explains why Intel is investing so much in this kind of studio. “Hollywood is about telling stories. This enables people to tell stories in immersive ways. Hollywood is on all platforms.”
“I’m very interested in how you mix reality with virtual reality. That creates really interesting opportunities for storytelling.”
Infinite Retina’s CEO, Irena Cronin, and its Chief Strategy Officer, Robert Scoble, meet with University of Southern California student Sagar Ramesh and get a tour through its MXR lab.
This is where students do Spatial Computing research and is only one of such labs at USC. Here he walks us through his student project, called Ollie, which lets VR wearers animate things in a virtual world in VR.
He let Scoble’s son, Ryan, play it and he discovers it along with you thanks to our new 360-degree camera. You can sign up to get the beta for free at http://ollievr.com.
Infinite Retina’s Chief Strategy Officer, Robert Scoble, has an 11-year-old son, Milan, who is autistic.
Now, he isn’t the kind of autistic that starts companies. He is going to have a tough time adopting to the “normie” world (what we call neurotypical people). He can barely talk. He isn’t always good at paying attention crossing roads. He will have a tough time holding down a job, particularly one that requires communication with other human beings (which will be most of them, particularly since the world is turning many jobs that he will be able to do into robot-run services). At Infinite Retina we have a bigger dream. What if, people like him (and he isn’t facing as difficult a challenge as many others) could be helped by technology?
We see a way, and so does a little company working away at BoostVC, which we see as the spiritual center for VR/AR/AI community.
That’s a startup accelerator located in San Mateo and was started by Adam Draper, the son of one of Silicon Valley’s most famous investors. His father, Tim, invested in tons of startups from Tesla to Hotmail, along with his firm DFJ (which recently changed its name to Threshold. The company is BehaviorMe.
Here Robert sits down with Andy Chavez and Annie Escalante joins via videoconference.
In the video, they explain how VR is being used to help autism therapists provide ABA therapy to kids with autism. They dig into what ABA therapy is, and how it’s currently applied. Today their system is being used by a few therapists who are seeing good results. Why? Because these kids take to technology, particularly technology like VR very quickly. Why? They usually are amazing at visual processing. Some autistic kids, in fact, are so good at visual processing they go on to have amazing careers, like Temple Grandin did developing cattle handling facilities.
But both we, and the team at BehaviorMe, see far more opportunity as new Spatial Computing devices come out (we use the term “Spatial Computing,” which means computing you can move around in, because there are many devices coming that will go far beyond the VR and AR devices we currently have and will include AI aspects as a norm).
In the interview we explore some of the ways autistic kids could be helped in the future with these kinds of technology. In fact, when Irena Cronin and Robert Scoble first were thinking of starting a company, he told her this is why he cared so much about Spatial Computing and the industry that quickly is forming to build and support it. We see how headset-based computing will greatly help his son have a productive life, from showing him how to do complex tasks to reminding him of basic things that we take for granted, like teeth brushing to finding him a ride to job or school via something like Uber or Lyft or a self driving car. Soon we will be wearing similar glasses, but for him those devices might be life saving.
Our team at Infinite Retina has been meeting with dozens of companies and scouring social media for interesting news about the industry. We retweet the best on our Twitter account at https://twitter.com/infiniteretina, but every month we’ll bring you a newsletter with the top things we are seeing in the industry.
Microsoft’s unknown weapon in battle for enterprise customers: the Azure Kinect DK camera
While most of the industry goes nuts over new VR headsets coming from Oculus or Valve, we’ve seen dozens of enterprise companies choose to develop for Microsoft’s HoloLens not just because of the new HoloLens 2 headset, but because of its unsung partner: a new 3D sensor.
“It’s a big deal,” Dirk Schart, President at Re’flekt America Inc., told us about the new Microsoft Azure Kinect DK camera. His company builds many augmented reality apps for companies like Audi, Daimler, and others.
Microsoft’s Jesse McCulloch, program manager on its mixed reality team (think HoloLens), gave Robert Scoble, Infinite Retina’s Chief Strategy Officer, a look at the new Microsoft Azure Kinect DK camera.
This product has had an effect on enterprise customers, Schart told us, even before it has arrived, because it makes possible a new kind of enterprise application that simply wouldn’t be possible with only head-worn displays. For instance, McCulloch told Scoble about a supply chain warehouse that’s using the camera aimed at the backs of trucks that are being loaded by robots. The camera sees the truck being loaded in 3D, and its AI capabilities categorize each package being loaded. A cloud-based app then ensures that the truck is being loaded properly, and tells robots how to place new packages in.
This camera has about twice as many data points as the 3D sensor in a new iPhone and it sees in 3D up to eight meters away, far further than the iPhone sensor can see. It also enables new kinds of computer vision-based applications. When McCulloch gave us a demo we saw various objects it saw were different colors and had different tags on them. All automatically added in real time.
Schart said he’s looking at it like a new kind of “eye” on a factory floor. Another company we are working with is building a surgical-assistant system. They will use such a camera over a person going through surgery. It ensures a constant data field that is stably located as doctors and nurses move around the room.
INFINITE RETINA VIEWPOINT
The “drought” is almost over. That’s what IDC just reported: https://www.businesswire.com/news/home/20190328005623/en/Augmented-Reality-Virtual-Reality-Headsets-Poised-Significant#.XJ0J_NXGKB0.twitter
As we meet with companies in the Spatial Computing space (which is really a big tent which includes everything from augmented reality to virtual reality to artificial intelligence and other technologies that run autonomous cars and smart cities; we think of it as all the software and hardware that allows you to compute and move through a 3D world). It’s clear that some are struggling to make revenues grow and profits are even harder to attain, while most have not even reached the revenue stage. Lots of companies are looking to fix that later this year when new headsets and supporting apps arrive that most expect will be popular with both enterprises and end consumers. The idea of Oculus Quest has brought the energy back to industry parties at conferences like GDC, which we recently attended. Add in Microsoft HoloLens 2 and Magic Leap and you can see excitement about the future of this industry is well justified.
That said, so few companies, particularly the startups we track, have found ways to monetize and find a substantial number of users. Here’s a list of more than 4,000 companies we are tracking in the market: https://twitter.com/Scobleizer/lists/all-orgs-in-spatial/members
Part of that is users just aren’t that interested in buying the headsets that are currently on the market. Congrats to Sony on selling more than four million, but, really, to be considered mainstream we have to see sales in many tens of millions and until a headset gets to ten million sold making business models work will remain extremely difficult. Another more important aspect than hardware issues is the fact that distribution for experiences, especially for VR, has not been figured out on an efficient and massive scale. Fragmentation and lack of information still rule.
We are just like everyone else, trying to hold back our excitement. We’ve been burned so much before. Our muted expectation does not keep us from pitching journalists on new stories, from expecting sales to close, and from building relationships with VCs that might really matter next year as this industry starts to see exponential growth again.
It’s time to not just get excited, but to pour on the gas and spend a lot more time showing Spatial Computing to people who haven’t seen it yet.
Recently we attended the first VR/AR Association meeting in Silicon Valley and heard about WebAR. The video of that meeting is up here: https://youtu.be/j9hdVqLZot0
But what we saw is not just about how important WebAR is. Think about it, if you walk into a store, do you want to load an app just to look at some 3D scans of products in that store? No, you would just want to hit a Web page and have them pop up on either your phone, or, in the future, on your Spatial Computing glasses.
We also heard that Amazon is betting big on WebAR and its Sumerian tool is key to helping merchants, and other users who need lightweight “instant on” augmented reality experiences.
Developers should pay attention to this tool for that reason, and, therefore, should pay attention to General Manager who runs the Sumerian team: Kyle Roche.
Here’s a video with Kyle from earlier this year on the Amazon Web Services YouTube channel: https://youtu.be/d2H5AmZaoCQ
He was the cofounder and CEO of 2lemetry, an Internet of Things startup, before joining Amazon when it acquired that company back in 2015.
Try Sumerian yourself at https://aws.amazon.com/sumerian/ and check out some projects from Sumerian Contest Winners, which were announced a few weeks ago: https://www.thevrara.com/blog2/2019/3/13/congratulations-to-amazon-sumerian-challenge-winners.
Welcome to the launch of our Spatial Computing Agency, Infinite Retina, which offers consulting services and research, a new video series, a new newsletter, a new podcast series, new industry databases (coming later this year), and a new book, coming in 2020. Along with our consulting services aimed at helping companies and entrepreneurs build new Spatial Computing projects and getting them funded and launched, we are very apt to helping everyone truly understand what the new paradigm of personal computing will be.
As we searched Google for the words “Spatial Computing,” we realized that a new definition was needed. This fourth paradigm of personal computing is coming into clearer focus and many of the definitions that were written a decade ago need a fresh look through the frame of 2019.
We need a new definition as Microsoft HoloLens or Magic Leap’s ML1 are giving us tantalizing glimpses into the near future, with the technology underneath providing a raft of new capabilities including new kinds of cameras (for example, the new Spectre camera, released on iPhones last week, gives us a taste of how Machine Learning can be used to bring new capabilities even to existing cameras). We are seeing Spatial Computing showing up in everything from June Ovens, Tesla cars, to Skydio autonomous drones that cannot just fly above us, but pick their own flight paths through complex environments. And we could go on for many thousands of other examples of products and companies using these new technologies.
Also, a new definition is needed because Spatial Computing isn’t just for humans. Magic Leap, Artie, Fable, and others, have virtual beings interacting not just with the humans talking to them, playing with them, and working with them, but their systems have situational awareness of the new AR Clouds, and, even of other virtual beings and robots in these new environments. This is pretty crazy science-fiction kind of stuff, except that as we close in on the beginning of the 2020s, it is becoming real and will have deep implications on all corporations and humans.
To underscore this development of virtual beings, we visited with the founders of Artie, who are pioneers in this area. In our video interview, they detail their company’s plans, challenges, and business imperatives.
What are the technologies of Spatial Computing?
- 3D and camera sensors to see the world.
- Machine Learning and even more advanced AI to understand that world.
- Systems that track both the real world, how it is changing, and the new “Mirror World” (we call them AR Clouds) and how they are changing, and, how the two fit together.
- Prediction engines that predict what the next action might be of both humans and machines.
- Optics and new kinds of displays to present a view of all this.
- Sensors and haptics to enable humans to better interact with, be studied by, and control this new world.
- New AI silicon chips that drive these engines.
- New situational awareness engines that fuse all this data together and enable virtual beings that interact with all of this.
- New content, entertainment, and games developed for this new world.
- New social networking and collaboration that happens in this fourth paradigm.
You can see this in our interview with the head of Niantic’s Augmented Reality efforts, Ross Finman. If you don’t know Niantic, this is the company underneath Pokémon GO, which is the most-used example of Spatial Computing we know of to date. In this video he details how he sees this new industry and its uses in gaming, as his teams put the finishing touches on a new Harry Potter game that will come later this year. Expected users? Hundreds of millions, most of whom will see a new form of Spatial Computing on their mobile phones for the first time.
Spatial Computing is the future of the personal computing industry, and will aggregate literally everything else into it. That includes AI, including Machine Learning and Natural Language Processing, Computer Vision, Augmented Reality, Virtual Reality, and all other apps that support the creation of and the maintenance of a digital 3D world, with the UX of the Internet to undergo major changes in the near future. All will be displayed in the optics of the Spatial Computing glasses you will wear, or new kinds of Holographic displays that will appear over the next decade on automotive windshields, new kinds of displays like the Looking Glass Display, and even contact lenses.
The definition: Spatial Computing is a series of technologies that can “see” and map the real world, understand it, and predict the next moves of objects within it, while doing the same for digital worlds that may or may not have ties to the real, physical world, all the while helping humans and/or machines navigate either.
While Infinite Retina really cares about AI, Augmented and Virtual Reality, and our partners are viewed as early adopters and valuable members of the community of people and companies that build such, we are seeing that calling this new world “XR” or “Mixed Reality” or, “Immersive” isn’t as satisfying as calling it “Spatial Computing.”
Why? Are the 3D sensing systems, like LIDARs that are being built into cars and glasses, “Immersive”? And what does “Mixed Reality” really mean? Finally, XR, and Mixed Reality come with Black Mirror baggage that Spatial Computing does not. These technologies should not be seen as scary, but should be seen as major new capabilities designed to help humans, not hurt them (although there certainly are downsides, and we will report on, and help corporations navigate the rough waters presented by them).
Irena Cronin, CEO of Infinite Retina, and Robert Scoble, Chief Strategy Officer, have visited many startups in this space and are working on a database that already has 5,000 companies in it, and our advisory board represents people working in many parts of the Spatial Computing industry from camera and sensor research to entertainment and cloud expertise.
Oh, and follow the money: we’ve seen there are tens of billions being invested in Spatial Computing by companies like Facebook, Magic Leap, Apple, Microsoft, HTC, and many others that we are under NDA with. We’ve never seen a spending spree in the tech industry like this, and it’s happening because of the promises of making computing far more personal, immersive, easier to use, and useful.
Our thesis is that for the next 24 months the first places you’ll see successful deployments of Spatial Computing are mostly gaming or enterprise uses, like surgery, logistics, mining, architecture, and car design, training, and many more. While we love playing games like Rec Room and Beat Saber in Virtual Reality, or Wonderscope or Pokémon GO in AR on our phones, we see a much bigger need as bigger companies, along with startups, bring new capabilities to market.
Even if you look at games, you’ll see that the people who fund a new round, or help them get noticed or get talent, are quite different than, say, the group of people and firms who brought us Airbnb, Spotify, or Instagram. We are building relationships with this new group and building new services around them, to bring their knowledge to new companies and projects that will bring new things to market.
Spatial Computing, though, isn’t just a way to make new companies or help older ones find new ways to cut costs or increase revenues. To us it’s personal. Robert Scoble’s son, Milan, is autistic and Robert is expecting these technologies to greatly increase Milan’s engagement with, and enjoyment of, the world. We talked with the head of innovation at drug company Pfizer and she told us that the company is doing research on Augmented and Virtual Reality on not just autism, but depression, pain, Alzheimer’s, and other health concerns. We’ve also talked with doctors who are using light to increase the perceptive abilities of athletes and others.
As these technologies become common we see huge changes for human beings. Some negative, like truck drivers who will probably lose their jobs to autonomous trucking, but many positive (the same trucks will have fewer wrecks and will use less energy and deliver things faster, and these technologies will retrain those same truckers much faster than educational technology has ever been able to do). These will assist us in living our lives, and, even present new wonderous entertainment capabilities.
We named our podcast, video series, and this newsletter “Spatial Computing Catalyst” for a reason: a rising tide lifts all boats, and our goal with all of these is to shine a light on the industry that’s building the fourth paradigm of personal computing, even as we work to build a series of services and, soon to come, research reports. We believe it’s one that will improve all of our lives.
We hope you do too. (Aside, this photo was from our first team meeting, held in the Palo Alto Apple store. Why? Infinite Retina got its name, in part, from Apple’s name for high-resolution screens. Spatial Computing will bring us all as many virtual high resolution monitors as we want, and we wanted to start our thinking about what Apple will do in this space – something that we’ll soon see an answer to as well).
– Irena, Robert, Marcelo, and Sam
Our CEO, Irena Cronin, and our Chief Strategy Officer, Robert Scoble, introduce themselves, and will regularly talk about the industry. Listen and subscribe here: https://infiniteretina.com/podcast/
VIDEOS WITH BUSINESS LEADERS
As part of our ongoing research and curiosity about the industry, We are doing video interviews with Spatial Computing entrepreneurs here: https://infiniteretina.com/videos/
BUSINESS LEADERSHIP ON SOCIAL MEDIA
Infinite Retina has already built more than 40 Spatial Computing Twitter lists: https://twitter.com/InfiniteRetina/lists, with more to come! Every week we watch those, along with tens of thousands of other people and companies who are doing Spatial Computing work, and we will present the best of business leadership here from social media each week.
Twitter list of the week: Spatial Architects. People and companies who help architects design better buildings using Spatial Computing technologies: https://twitter.com/InfiniteRetina/lists/spatial-architecture
Relevant Twitter posts:
Tipatat points at an interesting new book “The History of the Future” that takes us inside Facebook and its development, and acquisition of, Oculus Rift and Mark Zuckerberg’s AR/VR strategy.
Stanford’s VR lab links to a Wired article that presents new research showing that executives can be retrained to be more empathetic in VR.
Scott Stein at CNET (who did the best reporting about HoloLens 2 last week) has an in-depth interview with Alex Kipman, who runs Microsoft’s HoloLens group. In it he discusses his strategy for bringing HoloLens 2 to market, and what it might mean for Spatial Computing’s future.
Karl Guttag posts an in-depth review of HoloLens 2. He is an optics engineer, so he comes at the world explaining nits and field of view better than other analysts.
Niantic’s founder/CEO John Hanke, shows off its new multi-party AR platform, and says “This is one of those paradigm changes that happen maybe once a decade.”
https://twitter.com/NianticLabs/status/1100074685927174153 has a video of an interesting demo.
Charlie Fink shares part of his new book which covers Enterprise AR Use Cases.
AR Cloud pioneer Matt Miesnieks (cofounder and CEO of 6D.AI, one of several AR Cloud companies, and Advisor to Infinite Retina) gets bored on a flight and does an interesting Q&A about Spatial Computing.
Magic Leap announces its Indie Creator Program’s contest winners and it’s an interesting group of startups to watch.