GSoC 2017: Log #4

After resolving the largeVideo transition problem, the next step is to build a toolbar for micro window. I used Jitsi-meet’s ExternalAPI as the interface between the main Jitsi-meet iframe and the Micro mode.

Selection_057

JitsiMeetExternalAPI class creates a Jitsi-meet iframe on initialization, and provides several methods to control its components. The most basic features I first added to the micro mode are ‘mute-audio’, ‘mute-video’ and ‘hangup’.

Workspace 1_061

Once the toolbar buttons in micro window are pressed, it sends message to the Jitsi-meet iframe through Electron’s IPC channel, and then JitsiMeetExternalAPI toggles audio and video components accordingly.

Whenever the micro window pops up, the main window has to send the media status to the micro window, whether user’s audio or video is muted. For now I used a ‘hacky’ way, which is to access the ‘audio-mute’ button and ‘video-mute’ button in the iframe and check whether their className includes ‘toggled’ attribute. The main window sends the video status through the IPC channel, and the micro window switches its toolbar button layout accordingly. Since accessing an iframe’s DOM element directly is not really a good practice, a better way would be using the JitsiMeetExternalAPI interface instead. It can provides some ‘get’ methods whether the video or audio inside the Jitsi-meet iframe is muted.

 

GSoC 2017: Log #3

Previously, I managed to transfer the conference video from the main BrowserWindow to micro BrowserWindow using RTCPeerConnection API. However, one problem emerged was that it does not react to the dominantSpkearChanged event, which causes the transition of the largeVideo to another speaker in the Jitsi-meet application. This problem caused immense amount of trouble because it is so hard to solve without directly accessing the Jitsi-meet’s components.

At first, I thought of using importing Jitsi-meet’s APP object and then listen to the dominantSpeakerChanged event, but I got a feedback from the mentor that it is not a good design to directly access the components inside the iframe. So I am limited to work outside the iframe.

Another method recommended was to use HTMLMediaElement.captureStream() API, that returns a MediaStream object which is streaming of a real-time capture of the content being rendered in the media element. Surely, this seemed to be the ideal solution except it does not work in Electron BrowserWindow. The largeVideo DOM object does not has a “src” attribute, instead it has a “srcObject” attribute which is the MediaStream received from the current dominant speaker. Unfortunately, Chromium does not allow using captureStream() method on a video without “src” attribute, and throws the following error.

Uncaught NotSupportedError: Failed to execute 'captureStream' on 'HTMLMediaElement': The media element must have a source.

https://chromium.googlesource.com/chromium/src/+blame/5e1a7b0dd27f9eaed596106ee2726cab03df07c9/third_party/WebKit/Source/modules/mediacapturefromelement/HTMLMediaElementCapture.cpp#24

This has been resolved in the Chromium version 59.x.x.x, but even the latest version of Electron, v1.7.5 beta uses Chromium version 58.x.x.x. So there is no legal way to use HTMLMediaElement.captureStream() on the Jitsi-meet’s largeVideo in Electron environment.

One work around I tried was to make a canvas copy of the largeVideo, and apply HTMLCanvasElement.captureStream().

/**
 * Create a copy of HTMLvideo in HTMLcanvas for
 * Get reference of the video, width and height as parameters
 * Return canvas object
 */
function copyVideo(video, width, height, frameRate) {
  let canvas = document.createElement('canvas');
    canvas.width = width;
    canvas.height = height;
    let ctx = canvas.getContext('2d');
    setInterval(function() {
      ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    }, frameRate);
    return canvas;
}

This approach worked and I was able to capture the Jitsi-meet’s largeVideo onto a canvas element, which reacts to the video transition event properly.

I really hoped this canvas approached solve the video transition problem for once and for all, but it did not. Once the window get minimized, the canvas stop capturing the video, probably because for most of the cases there is no point of capturing visual data when the window itself is not visible.

After some desperate researching and all alternatives failed, I posted a question in StackOverflow and got an unexpected solution for this long dragged problem.

https://stackoverflow.com/questions/45156701/html-canvas-drawimage-when-window-minimized

It turned out that HTMLVideoElement.ondurationchange event can detect the change in srcObject of a video element. In this way, I don’t have to access the Jitsi-meet iframe’s App component and still be able to react to the dominantSpeakerChanged event. After discovering this, the rest of the process was very easy; whenever ondurationchange event is emitted, the main window remove the existing MediaStream object from the p2pChannel and attach the new MediaStream object.

I have been struggling so long to solve this problem yet the answer was somewhat disappointingly simple. One thing I realized was that WebRTC is a fairly new technology and a lot of its parts are under development. Even the addStream() method I used to transfer the MediaStream object is deprecated, and was recommended to use addTrack() method instead. However, addTrack() method is not supported in Chrome, so I was forced to use a deprecated method. This is not the only case of using experimental features in WebRTC, so I foresee lots of upcoming maintenance works as some of the features become unsupported in near future.

GSoC 2017: Log #2

Previously, I have set up a RTCPeerConnection between the main jitis-meet window and the micro window for the video transmission.

However, it hard coded a lot of stuffs and honestly speaking, the code design was a total mess. There were both main window side’s code and micro window side’s code in one page, so it was vulnerable to unintended global variables usage and namespace conflicts. I decided to refactor the code into OOP-style one, so that it can be used in more general case. It is sort of a wrapper for the webkitRTCPeerConnection, suited for the inter-Electron BrowserWindow communication which does not require a STUN server for signalling.

API

Class: WindowPeerConnection

Parameter

  • windowName: the BrowserWindow’s name in main process

The WindowPeeerConnection wraps around webkitRTCPeerConnection class, which on construction in a renderer process, sets up a IPC listeners that receive message and data from other windows. All the message are relayed through the main process, and data are serialized before transmitted.

Method

  • attachStream ( streamToSend ) : attaches a MediaStream object on the peer connection channel, which can be transmitted by sendStream method. Currently, it can only attach one stream per connection.
  • removeStream ( ) : removes the MediaStream attached to this peer connection channel.
  • sendStream ( receiverName ) : sends the attached MediaStream object to the target window specified the string ‘receiverName’. receiverName refers to the name of the variable assigned to the target BrowserWindow in the Electron’s main process.

EventListener

  • onReceivedStream ( function(receivedStream) ): triggered when the WindowPeerConnection receives a MediaStream from a remote window.

 

Main Process: Data Relay Channel

A Javascript object that is called in the Electron’s main process. Since Electron does not support direct IPC between BrowserWindows, all messages and data must be relayed through the main process, using ipcMain. Each BrowserWindow is wrapped as a client in the data Channel.

Function

  • addClient ( client ) : adds a client on the data relay channel. A client is a Javascript object with the following format:

client= {window: BrowerWindow object,

name: "BrowserWindow's name"}

  • initChannel ( ) : initiates the data relay channel with the current array of clients. Clients can be furthered added after the channel is initiated.

 

Example

Main Processmain

Sender Windowwindow1

Receiver Windowwindow2

 

Currently, it only supports transmission of MediaStream, but it could be extended to general Data transmission later on. I might be going to publish this code in npm, since I could find anything similar.

GSoC 2017: Log #1

First of all, I need to make sure that the large video in the Main window’s Jitsi-meet iframe also appears in the Micro window. If I can’t even access the video elements in the Jitsi-meet iframe, I can’t proceed to anywhere.

There were three approaches I attempted:

  1. Import the video element directly from the render process, using ‘module require’
  2. Use createObjectURL to generate URL of the video element and pass it to Micro mode’s window.
  3. Instead of creating a separate window for Micro mode, change the layout of main window when the window is minimized.

After trying all three, I realize that none of them actually works. The most important problem I missed out was that each Electron BrowserWindow is an separate, independent process, and Electron does not provides a Inter-Process-Communication(IPC) feature for media elements.

The reasons each approach failed are:

  1. Node’s ‘module require’ does not support transferring of a HTMLMediaElement to another page. It became ‘undefined’ every time I attempted.
  2. Similar to the ‘require’ approach, the scope of URL create is limited within the same page. When I tried to receive the media element from the URL, I only got 404 response.
  3. This approach only works partially. I can restructure the layout every time the Micro mode is called, but that means the Normal mode and the Micro mode shares the same window. So when the user minimizes the (only) window, the Micro mode’s window gets minimized as well.

 

In the end, I ended up using webRTC’s RTCPeerConnection to set up a peer-to-peer connection between the main window and the micro window. By right, I was supposed to use a STUN/TURN server to do signalling between the main window and the micro window. But since they are both under the Electron’s main process, I used Electron’s IPC feature instead to pass necessary information for signalling.

After creating html and js files for the Micro mode, I added a module for the peer connection between the main window and the micro window.

The WebRTC connection is established in following order:

     Main Window                                                                Micro Window

1.  Sets up RTCPeerConnection

                                                                                     2. Sets up RTCPeerConnection

3. Attaches jitsi video stream to peer connection

4. Creates and sends connection offer

                                                                                            5. Receives the offer

                                                                                            6. Creates and sends answer

7. Receives the answer

8. Sends ICE candidate

9. Receives and adds ICE candidate

                                                                                      10. Receives the media stream

And then… Voila!

rtc

https://github.com/jitsi/jitsi-meet-electron/pull/12/commits/a95e12ec17630701f035b2e71d8c92d6ed4754c1

 

GSoC 2017: Log #0

This summer is for code. My proposal has been accepted to the Google Summer of Code(GSoC) 2017, so I would be working under Jitsi for next few months. I never thought I would be accepted as this was my first attempt to GSoC, but hell yeah.

GSoC

Project: Implementing ‘Micro Mode’ for Jitsi-Meet-Electron

URL: https://summerofcode.withgoogle.com/projects/#4757550437236736

This is a simple feature for Jitsi-Meet’s desktop app, which pops up a small video and set of GUIs when the user minimizes the main window. It is a similar one to the one in Skype, hope I can make it cooler than that.

So excited to start working on the project, and many thanks in advance to the mentors Hristo Terezov and Saúl Ibarra Corretgé who will be guiding(tolerating) me for the period of time. Hope I won’t fall behind.