# Depth estimation

VisionKit provides depth estimation capabilities.

Depth estimation is to obtain the distance information from each point in the scene to the camera in the image, which is called depth map.

# Definition of method

We present here two modes of depth estimation:

Visual pattern: (AndroidWeChat>= 8.0.37, iOS WeChat>= 8.0.38 From) Depth Estimation Using Cameras 。

AR mode: (AndroidWeChat>= 8.0.38 Simultaneous Base Library>= 2.33.0, iOS WeChat>= 8.0.39 Simultaneous Base Library>=3.0.0) Using a camera and IMU for depth estimation, output by near and far control of the nonlinear depth map, for virtual and real occlusion, currently support some models, support models list is equivalent to[6Dof- Horizontal AR-V2 Planar AR Interface Support List](https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/Visionkit /plane.html)。

# Visual mode interface

The visual mode interface provides two methods of use, one is to input a static image for depth estimation, and the other is to perform real-time depth estimation through a camera.

1. Static picture estimation

adopt[VKSession.detectDepth interface](https://developers.weixin.qq.com/miniprogram/dev/api/ai/Visionkit /VKSession.detectDepth.html)Input an image, and pass the[VKSession.on](https://developers.weixin.qq.com/miniprogram/dev/api/ai/Visionkit /VKSession.on.html)The interface listens to the depth map information, the pixel value in the map represents the current depth value, the darker the color, the closer it is to the camera, and the whiter the color, the farther it is from the depth.

Sample code:

const session = wx.createVKSession({
  track: {
    depth: {
      mode: 2 // mode: 1 - Use the camera2 - Manual input image
    },
  },
  gl: this.gl,
})
// In Static Picture Estimation mode, each tone detectDepth The interface will trigger once updateAnchors event
session.on('updateAnchors', anchors => {
  anchors.forEach(anchor => {
    console.log('anchor.depthArray', anchor.depthArray) // Depth chart ArrayBuffer data
    console.log('anchor.size', anchor.size) // Depth chart size, resulting in an array[Wide, high]
  })
})

// Need to call once start To initiate
session.start(errno => {
  if (errno) {
    // If it fails, it will return. errno
  } else {
    // Otherwise, return null, indicating success
    session.detectDepth({
      frameBuffer, // ArrayBuffer of the picture to be detected Data. The depth image RGBA data to be detected, 
      width, // Image width
      height, // Image height
    })
  }
})

2. Real-time estimation via camera

The algorithm outputs the depth map of the current frame in real time. The pixel value of each frame represents the current depth value. The darker the color, the closer it is to the camera, and the whiter the color, the farther it is from the depth.

Sample code:

const session = wx.createVKSession({
  track: {
    depth: {
      mode: 1 // mode: 1 - Use the camera2 - Manual input image
    },
  },
})

// Need to call once start To initiate
session.start(errno => {
  if (errno) {
    // If it fails, it will return. errno
  } else {
    // Get the information for each frame
    const frame = session.getVKFrame(canvas.width, canvas.height)
    // Get depth map information for each frame
    const depthBufferRes = frame.getDepthBuffer ()
    const depthBuffer = new Float32Array(depthBufferRes.DepthAddress)
    //Create rendering logic, Transfer the array value to a texture and render to the screen
    Render()
  }
})

# AR mode interface

See details[V2 + Virtual and Real Occlusion in Horizontal AR](https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/Visionkit /plane.html)

# Real Time Estimation Output Dxplaination

# 1. Effect display

# 2. Depth Estimation Key Points

Deep Real Time Key Point Output Fields include

struct anchor
{
  DepthAddress, // The address of the depth map ArrayBuffer, Usage as new Float32Array(depthBufferRes.DepthAddress) 
  width,    // Returns the width of the depth chart
  height,      // Returns the height of the depth chart
}

# Application Scenario Examples

1. Special effects scenes.
2. AR games and applications(The following is an example of real AR occlusion)。

# Program Examples

Visual mode:

1. [Real-time depth estimation capability using reference](https://github.com/WeChat mini-program /miniprogram-demo/tree/master/miniprogram/packageAPI/pages/ar/depth-detect)
2. [Static image depth estimation capabilities using reference](https://github.com/WeChat mini-program /miniprogram-demo/tree/master/miniprogram/packageAPI/pages/ar/photo-depth-detect)

AR mode:

1. [V2 plane + virtual and real occlusion](https://github.com/WeChat mini-program /miniprogram-demo/tree/master/miniprogram/packageAPI/pages/ar/plane-ar-v2-depth)

# Demo Experience

Visual mode:

1. Real-time depth estimation capability, in the Mini Program exampleInterface - VisionKit Vision Capabilities - Real-time Depth Map DetectionExperience in.
2. Photo depth estimation capability, in the Mini Program exampleInterface - VisionKit Visual Capabilities - Photo Depth Map DetectionExperience in.

AR mode:

1. In the Mini Program exampleInterface - VisionKit Visual Capabilities - Horizontal AR-v2 - Virtual OcclusionExperience in.