# VoIP Video Streaming Guide

Weixin Mini Program Voip's video stream defaults to H264 encoding stream, from the device side, divided into device upstream stream and WeChat downstream, resolution is currently the maximum 640 x 480.

For the upstream stream, it is device capture - > device encoding - > network transmission - > WeChat reception - > WeChat decoding - > Weixin Mini Program rendering. For downlink streams, it is WeChat Collection - > WeChat Encoding - > Network Transmission - > Device Reception - > Device Decoding - > Device Rendering.

It is important to note that the original resolution of the video stream captured by the WeChat client using the camera is 640x480, and its direction is not consistent with the physical direction, so for RTOS and Linux devices, when the stream is received, it needs to be reprocessed to render correctly.

# 1. Android devices

Developers of Android devices don't have to worry about the direction of video streams.WMPF rotates the received video stream, Weixin Mini Program rendering the output (objectFit is fill).

# 2. Linux / RTOS devices

This type of device cannot run WMPF and uses the SDK directly for receiving and sending video streams.

# 2.1 Device sends video streaming

Local acquisition of the mobile phone camera rotates the inverse clock in the 90 degree direction shown in the above diagram, but this is not generally the case with embedded devices.So when a camera capture stream on an embedded device is sent directly using the SDK, the WeChat end Weixin Mini Program Voip plug-in renders a rotating image without using setUIConfig to configure the rotation.

There are two solutions:

• The video stream that rotates the embedded device camera is a video stream that turns 90 in reverse clock time.
• On Weixin Mini Program the setUIConfig uses plug-ins to configure The opposite rotates in a direction of 270.

How does the SDK end send a video stream to enable the Weixin Mini Program end to display images as early as possible?

• H264 / H265 streams need to be sent after the camera_device- > open_stream call implemented by the developer has completed.

• Ensure that the first few lines are sent as information lines and key lines, in the following order:

  sps -> pps -> i-frame -> p-frame -> p-frame -> ......

• Do not use streams with B frames

• You need to start sending when the state is TALKING, otherwise the previous frame may be discarded and the p-frame will fail to send, and only the next frame will succeed.

It is important to note that the video streaming resolution, long and wide values sent by the device need to be aligned with 8.

# 2.2 The device receives video streaming

The device receives a WeChat video stream captured and encoded by the client using the camera. When captured by the camera on Android and IOS, the original stream is a stream rotating 90 clockwise, which is exactly what the device receives.

There are two solutions:

• Recommendation: After receiving the stream, the embedded device rotates 90 clockwise at the output.
• Use the > = 8.0.54 version of the WeChat Guest, use the > = 2.4.5 version of the front-end VoIP plug-in, and use the SDK version that supports detailed subscription parameters.For detailed methodology, see 2.4

# 2.3 Video streaming resolution

Weixin Mini Program The resolution of Voip video is not fixed, it changes with the quality of the network.

• Device uplink video streaming: resolution should not exceed 1080p. The resolution is allowed to change during the call, but it is necessary to send SPS, I frames, etc. The typical resolution is 320 x 240, 640 x 480.
• Downlink video stream: WeChat downlink video stream starts from 320x240 / 240x320 resolution, and then increases to 640x480 / 480x640 according to the network situation.

For WeChat downlink video streams, some RTOS devices do not support the ability to handle resolution changing streams.Developers can use the 2.4 method to fix the resolution.

# 2.4 WeChat Video Coding Customization Method

For RTOS or Linux devices, when the device requires a very demanding h264 / h265 codec sent by WeChat, you can use the following methods to customize WeChat coding behavior to obtain custom video streams.

We need to understand four parameters:

• The encodeVideoFixedLength encodings have long edges fixed, not changing with network quality, preferably 320, 480, 640
• Encodevideorotation encoded video rotation direction.1: Give out a positive flow. 2: Keep give out a rotating flow
• EncodeVideoRatio Video ratio, width / height * 100
• EncodeVideoMaxFPS Maximum FPS for video.Smallest 8, maximum 15.

These parameters need to be set in two places at the same time:

• WeChat end : When the current VoIP plug-in version > = 2.4.5, the plug-in will parse out these fields from the query parameter and set them into WeChat.No matter what kind of call interface has query parameters, you can set these fields correctly in the query.
• SDK side : Use the SDK version that supports these parameters and set them up correctly.The settings entry on the SDK side is subscribe_video in wx_init_config_t_length、subscribe_video_rotation、subscribe_video_ratio、subscribe_video_maxfps

Let's look at a case where a device wants to receive a normal flow of 240x320

• When calling, you can pass query parameters such as:

encodeVideoFixedLength=320&encodeVideoRotation=1&encodeVideoRatio=75

• SDK side, to config.subscribe_video_length、config.subscribe_video_rotation、config.subscribe_video_ratioAlso pass in the corresponding value.
• The front-end Weixin Mini Program recommends doing a WeChat version judgment logic, greater than 8.0.54 to use this feature.

Other resolutions can also be customized, but it should be noted that Weixin Mini Program VoIP scenariosWeChat The camera originally captures a stream with a resolution of 640x480 and a reverse clockwise rotation of 90 for a physical direction, which means that you cannot customize a normal orientation flow of 640x480. You can customize only a normal direction flow of 480x640.

# 2.5 Performance reference

# 2.5.1 WeChat Ring delay

• The wx_cloudvoip_client_call interface normally takes about 600ms. If running on an RTOS device, the return time of the interface will vary depending on the network side implementation.
• When the wx_cloudvoip_client_call interface returns successfully, the WeChat side will ring.Of course, the delay will be different for different phones. For iphone, also rely on apple's message push, for android, a variety of settings may also lead to abnormal phone messages, can refer to call reminder exception troubleshooting guide

# 2.5.2 Video streaming latency

The latency in the following scenarios has a lot to do with network quality, and the following data are based on normal network conditions.

• After the successful answer, the device first received the video stream delay of about 1s ~ 2s, on the end of different performance of mobile phones, different WeChat version, the time will have a little difference.

• After answering successfully, the delay of the first video stream received by the mobile phone is about 1s ~ 2s.Note that if the phone receives directly from a suspended notification, it will go through two processes: (1) open Weixin Mini Program and (2) join the call, where 1s ~ 2s refers to the (2) join the phone process. The time it takes to open a Mini Program is related to the performance of your phone.

• In a typical scenario, device capture video - > transfer to the cloud - > feed into the cloud SDK interface - > Weixin Mini Program end of the picture: The delay of the whole process is about 200 ~ 300 ms.