Better focus and better photos
You may have read something about a time-of-flight sensor and a 3D camera lately. This is also referred to as a LiDAR Scanner, which you also encounter in self-driving cars because it can be used to estimate distances properly. But what exactly is a LiDAR Scanner and what can you do with it?
- Suitable devices
- Explanation
- Possibilities
Which devices have LiDAR?
You can find the LiDAR Scanner in the following devices:
- iPhone 13 Pro (Max)
- iPhone 12 Pro (Max)
- iPad Pro 2021
- iPad Pro 2020
So it seems that this sensor is mainly reserved for the Pro devices. The iPad Pro 2020 was the first device to have these sensors and since the iPhone 12 series you can find it in the Pro models.
What is LiDAR?
LiDAR stands for Light Detection and Ranging. It is a technique that allows a device to scan the environment and thus calculate distances to objects. In self-driving cars, this is often used in combination with other technologies, such as radar.
LiDAR uses tiny invisible lasers. By reflecting off an object, the device measures how long it takes for the light to be reflected. Based on this, a calculation is made of the distance to an object. That is why this is also called time-of-flight (ToF): the time during which the light ‘flies’ through the air. NASA also uses LiDAR for Mars landings.
On the iPad Pro we find the LiDAR Scanner in the camera bump, right next to the two lenses. The LiDAR Scanner works together with the cameras, motion sensors and the fast processor to scan the environment, measure distances and recognize objects. The LiDAR Scanner in Apple devices can scan objects up to 5 meters away. Many lasers are sent and received every second, which means that many calculations are involved. The A12Z chip handles all those processes.
On the iPhone 12 Pro (Max), the LiDAR Scanner can be recognized by the black dot in the camera bump. Apple itself says: “The Pro series has an all-new LiDAR scanner, which can measure light distance and use information about pixel depth. This technology delivers faster and more realistic AR experiences as well as 6x faster autofocus in low light, sharpening photos and videos and reducing shooting time.” So it is not only useful for AR (as on the iPad Pro) but also for sharper photos and better image quality.
What can you do with the LiDAR Scanner? What are the possibilities?
As indicated above, the LiDAR Scanner on the iPhone is mainly used to determine depth faster when photographing. This produces sharper images, especially when shooting under moderate lighting conditions. The photo below is an example, taken by Apple itself.
Snapchat uses the LiDAR Scanner to use for special lenses. Lens Studio is made available for this. The scanner measures the environment so that the Snapchat lens can provide a more realistic view of objects and surfaces. By combining the power of the A14 Bionic chip and ARKit, you can make thousands of augmented reality objects appear in real time.
In the iPad Pro, Apple mainly uses the LiDAR Scanner to improve the augmented reality functions. If you have ever used an AR app, you will recognize that you have to calibrate the app before you can use the augmented reality functions. You then have to move your iPhone or iPad back and forth so that the cameras can scan the environment. Thanks to the LiDAR Scanner, this happens much faster: the environment is immediately recognized and the depths can be measured much more accurately.
The device with the LiDAR Scanner is also more aware of the environment. This means that you can, for example, place a virtual chair behind an object in the real world, such as a table. As a result, virtual objects blend much better into the real world, resulting in a much more natural appearance. AR objects also stay in place better while you walk around them and they take more into account the presence of real people. This means that you can walk between AR objects, without them all appearing in front of the person.
Apple’s own Measure app also makes full use of the new possibilities. Measurements are performed faster and you can now automatically measure someone’s height. There is also a new Ruler View, which allows you to make more detailed measurements. You can save all these measurements with accompanying screenshots, so that you can consult them later.
Other developers have also started working with the LiDAR Scanner. The IKEA Place app has a new Studio Mode that allows you to furnish entire rooms. The previous app let you place a single piece of furniture in your living room, but with the new advanced Studio Mode you can furnish your entire home. This can also come in handy in games. For example, the Apple Arcade game Hot Lava has received a new AR mode. In it, your own living room is transformed into a playing field full of lava.
All apps built with ARKit, Apple’s proprietary augmented reality app development tool, already benefit directly from the LiDAR Scanner. For example, with these apps it is no longer necessary to calibrate and scan the environment first. Developers of these apps do not have to do anything for this.
