Mini Program
取消
Development
中文
EN
取消
  • Guide

    • # VoIP Video Streaming Guide

    Mini Program Voip Of the video stream defaults to H264 Coding stream, from the device side, is divided into device upstream stream and WeChat downstream stream, the resolution is currently the largest 640x480。

    For the uplink, it is the device acquisition -> Device coding -> Network transmission -> WeChat receiving -> WeChat decoding -> Mini Program rendering. For the downstream, it's WeChat collection -> WeChat coding -> Network transmission -> Equipment receiving -> Device decoding -> Device rendering.

    It should be noted that the WeChat client uses the camera to capture the original resolution of the video stream for 640x480, and its direction is not consistent with the physical direction, so for RTOS and Linux Devices, which, after they receive the stream, require post-processing to render correctly.

    # 1. Android equipment

    Android The developer of the device does not have to worry about the direction of the video stream. WMPF Will rotate the received video stream, the Mini Program rendering output can be (objectFit for fill)。

    # 2. Linux/RTOS equipment

    This type of device cannot operate WMPF, directly using the SDK Receive and transmit video streams.

    # 2.1 Device Sends Video Stream

    The local capture of the phone's camera is rotated counterclockwise as shown in the image above 90 But embedded devices generally do not have this situation, so when the camera on the embedded device captures the stream directly SDK After sending out, WeChat end Mini Program Voip The plugin is not using the setUIConfig When configured to rotate, it will render the rotating screen instead.

    There are two solutions:

    • Configure the video stream of the embedded device camera to rotate counterclockwise 90 Of the video stream.
    • The use of plug-ins on WeChat mini programs [setUIConfig ](https://developers.weixin.qq.com/miniprogram/dev/framework/device/voip-plugin/api/setUIConfig .html) Configure the direction of rotation of the other party to 270。

    SDK How to send a video stream to make the Mini Program side display the image as early as possible

    • Need to be implemented in the developer's Camera_device->open_stream After the call is complete, you can send H264/H265 Stream.

    • Ensure that the first few frames are sent as information frames and key frames, and the order should be as follows:

      sps -> pps -> i - Frame -> p - Frame -> p - Frame -> ......

    • Need to be in the state of TALKING Before you start sending, otherwise the previous frame may be discarded p - Frame Failure to send, only to wait for the next cycle of information frame to succeed.

    # 2.2 Device receives video stream

    The device receives the video stream collected and encoded by the WeChat account using the camera. and IOS When captured by the camera, the original stream is rotating counterclockwise 90 Of the stream, the device receives exactly that.

    There are two solutions:

    • Recommended: The embedded device, upon receiving the stream, rotates clockwise at the output 90 Just...
    • Provide Mini Programs appid To WeChat Voip Team, open the ability to send non-rotating stream, after opening the WeChat client will first rotate the captured video stream to correct, and then send it out. It is important to note that this feature will have an impact on appid All equipment under (including WMPF equipment) Cause an impact

    # 2.3 Video stream resolution

    WeChat Mini Program Voip The resolution of the video is not fixed, it will change with the quality of the network.

    • Device uplink video streaming: resolution should not exceed 1080p。The resolution is allowed to change during the call, but it needs to be sent in time. SPS 、I Zhen, etc. The typical resolution is 320x240、640x480。
    • WeChat downlink video streaming: The downlink of WeChat video streaming is from 320x240/240x320 Resolution to start with, and then, depending on the network, will be raised to 640x480/480x640。

    For WeChat downlink video streaming, some RTOS The device does not support the ability to handle resolution change streams. Developers can provide appid To WeChat Voip Team, the ability to turn on fixed resolution (320 x 240/240x320)。 It is important to note that this feature will have an impact on appid All equipment under (including WMPF equipment) Cause an impact 。

    # 2.4 Performance reference

    # 2.4.1 WeChat ring delay

    • wx_cloudvoip_client_call Interface Normal Needs 600ms Left and right. If in RTOS Depending on the network side implementation, the return time of the interface will vary.
    • wx_cloudvoip_client_call After the interface returns successfully, the WeChat side rings. Of course, for different phones, the delay situation will be different. for The iphone, and also relies on apple Of the message push, for the Android, a variety of settings may also lead to abnormal phone messages, you can refer toCall Alert Anomaly Troubleshooting Guide

    # 2.4.2 Video Stream Delay

    The delay in the following scenarios has a lot to do with the quality of the network, and the following data are based on the normal situation of the network.

    • After receiving successfully, the delay of the device receiving the video stream for the first time is about 1s 2s Right or left, the time will be slightly different for different performance mobile phones and different WeChat versions.

    • After the successful answer, the delay of the mobile phone receiving the video stream for the first time is about 1s 2s Left and right. It should be noted that if the mobile phone is answered directly from the suspension notification, it will experience (1) opening the Mini Program and (2) joining the call. Two processes. Here's the 1s 2s Refers to (2) joining the call process. The time taken to open the Mini Program is strongly related to the performance of the mobile phone.

    • Typical scene, the device captures video -> Sent to the cloud -> To the cloud SDK interface -> WeChat Mini Program out of the picture: The delay of the whole process is about 200 300ms。