OpenVidu Flutter problem

[icelija]

Hello, I took this flutter-webrtc-demo app and tried to connect it to OpenVidu webrtc server because I need to have multiple users to be able to participate in the same call. I used https://github.com/OpenVidu/openvidu-android-app which works with OpenVidu server v1.5.0 (the latest is v2.7.0, and we set up our own instance with v1.5.0 at openvidu.hot-soup.com) and refactored Signaling class to work as this Android app works.

The problem is when I connect my Flutter app with Android app, the stream from Android app is visible on Flutter app but the stream from Flutter app is not visible on Android app and I can't find out why. I examined the logs from both apps and it seems to me that signaling with the server through websocket connection works in the same way, but something is wrong because stream from Flutter app is not visible on the other side.

Here is the link of Android app that is connected to openvidu.hot-soup.com: https://drive.google.com/open?id=1ln8rHhgUq8PEanoxT7ciYNHkcAZfeIjt

And here is the link of Flutter app connected to the same server: https://drive.google.com/open?id=1cXfNCG7UAvDZ0gN0FmmPpSpORZJo5f9x

Can you maybe take a look or do you have any idea why this doesn't work and the example connected to demo.cloudwebrtc.com server works great?

[cloudwebrtc]

Can you provide a more detailed log because I am not familiar with how OpenVidu works?

[icelija]

@cloudwebrtc I have provided two logs, one is from Android app (connected to the room with another Android app and stream is visible in both apps) and the other one is from Flutter app (connected to the room with Android app and stream is visible only in Flutter app from Android).

openvidu-android-log.txt
openvidu-flutter-log.txt

[cloudwebrtc]

I found this sdp in your flutter log and it seems to be invalid.

I/flutter (29830): onMessage: {id: 2, result: {sdpAnswer: v=0
I/flutter (29830): o=- 3757068764 3757068764 IN IP4 0.0.0.0
I/flutter (29830): s=Kurento Media Server
I/flutter (29830): c=IN IP4 0.0.0.0
I/flutter (29830): t=0 0
I/flutter (29830): a=msid-semantic: WMS
I/flutter (29830): , sessionId: t5md5q2isjjlh7hghirkut3hjs}, jsonrpc: 2.0}```

[icelija]

@cloudwebrtc I see, I think this was caused because I changed from this

  final Map<String, dynamic> _constraints = {
    'mandatory': {
      'OfferToReceiveAudio': true,
      'OfferToReceiveVideo': true,
    },
    'optional': [],
  };

to this

final Map<String, dynamic> _constraints = {
    'mandatory': {
      'offerToReceiveAudio': true,
      'offerToReceiveVideo': true,
    },
    'optional': [],
  };

I returned back to the dict with uppercased attributes and it seems that now "sdpOffer" in "publishVideo" that is sent and "sdpAnswer" that is returned is ok, but still there is no stream on the other side.

Here is new log from Flutter app:
openvidu-flutter-log.txt

[icelija]

@cloudwebrtc can you please take a look, I fixed the issue that caused wrong "sdpAnswer" in the log you mentioned but still no stream is shown on the other side?

[huulbaek]

@icelija Would you be able to share your work again? I am about to implement in Flutter too.

[icelija]

@huulbaek here you go new links from the projects that were linked in the first post:

Link of Android app that is connected to openvidu.hot-soup.com:
https://drive.google.com/file/d/1BMFQJJydd9AYQx0bqanxGu1x9UF40SqF

Link of Flutter app connected to the same server:
https://drive.google.com/open?id=1SYscclsNlyx6lGMMEcp55XUsL2zQrgs8

I didn't work anymore on it so the Flutter project still doesn't work properly.

[huulbaek]

I got it working ok. I made a several changes but mainly it was missing a local stream publish in _createLocalPeerConnection():

_localPeerConnection.addStream(_localStream);

[gvnramu]

Hi @huulbaek, I'm new to flutter and openvidu and I'm trying to integrate openvidu in the flutter. could you please share with me the working sample at gvnramu@gmail.com. Thanks in advance

[huulbaek]

I still need to fix some kind of race condition because it only works sometimes.

[rt16]

Hey @huulbaek can u please share the source code or a working sample for the project .I am new to flutter and im trying to integrate the openvidu in flutter.

[huulbaek]

I will try to make a sample when I have some spare time in the future. It will probably be after all this Corona business is over.

[cyb3rcod3]

@huulbaek

I got it working ok. I made a several changes but mainly it was missing a local stream publish in _createLocalPeerConnection():

_localPeerConnection.addStream(_localStream);

I will try to make a sample when I have some spare time in the future. It will probably be after all this Corona business is over.

Share more info, please.
I play with this:

Link of Flutter app connected to the same server:
https://drive.google.com/open?id=1SYscclsNlyx6lGMMEcp55XUsL2zQrgs8

..with flutter-webrtc 0.2.8, openvidu-ce (openvidu-server 2.14 and kms 6) without luck.
No video stream output from flutter Android app and no remote stream received on browser client as video-call example.
Start test calling wss, create session, request token, and make "joinRoom" from flutter app, remote desktop client show new partecipant but no audio and no video in stream, ( i add "_localPeerConnection.addStream(_localStream);", but it doesn't seem enough to fix.

Log attacched
test001_app_browser.log

[huulbaek]

@neoxxx85 Do you mean OpenVidu 2.14?

Anyway, I haven't had time to clean up my own code, so it is very much tied up to specific code in my project. Here is my main code though:

import 'dart:convert';
import 'dart:async';
import 'dart:io';
import 'package:flutter_webrtc/webrtc.dart';
import './utils/rtc_ice_candidate.dart';
import 'package:punkr/components/rtc/models/participant.dart';

enum SignalingState {
  CallStateNew,
  CallStateRinging,
  CallStateInvite,
  CallStateConnected,
  CallStateBye,
  ConnectionOpen,
  ConnectionClosed,
  ConnectionError,
  CallStateWaiting
}

class JsonConstants {
  static const String value = 'value';
  static const String params = 'params';
  static const String method = 'method';
  static const String id = 'id';
  static const String result = 'result';
  static const String iceCandidate = 'iceCandidate';
  static const String participantJoined = "participantJoined";
  static const String participantEvicted = "participantEvicted";
  
  static const String participantUnpublished = "participantUnpublished";
  static const String streamPropertyChanged = "streamPropertyChanged";
  static const String sendMessage = "sendMessage";

  static const String participantPublished = "participantPublished";
  static const String participantLeft = "participantLeft";
  static const String sessionId = "sessionId";
  static const String sdpAnswer = "sdpAnswer";
  static const String joinRoom = "joinRoom";
  static const String metadata = "metadata";
  static const String publishVideo = "publishVideo";
  static const String onIceCandidate = 'onIceCandidate';
  static const String receiveVideoFrom = 'receiveVideoFrom';
  static const String endpointName = 'endpointName';
  static const String senderConnectionId = 'senderConnectionId';
  static const String connectionId = 'connectionId';
  static const String leaveRoom = 'leaveRoom';
  static const String streams = 'streams';
}

typedef void SignalingStateCallback(SignalingState state);
typedef void StreamStateCallback(MediaStream stream);
typedef void OtherEventCallback(dynamic event);

class Signaling {
  WebSocket _socket;
  String _url;
  String _token;
  String _session;
  String _turnUsername;
  String _turnCredential;
  int _realUserId;
  int _realGroupid;
  int _internalId = 1;
  String _userId;
  String _endpointName;
  Timer _timer;
  MediaStream _localStream;
  Map<String, RemoteParticipant> _participants = <String, RemoteParticipant>{};
  Map<String, String> _participantEndpoints = <String, String>{};
  RTCPeerConnection _localPeerConnection;
  bool _localPeerConnectionHasRemoteDescription = false;
  SignalingStateCallback onStateChange;
  StreamStateCallback onLocalStream;
  StreamStateCallback onAddRemoteStream;
  StreamStateCallback onRemoveRemoteStream;
  OtherEventCallback onParticipantsRemove;
  OtherEventCallback onParticipantsJoined;
  OtherEventCallback onParticipantsStreamUpdate;
  OtherEventCallback onMessageReceive;
  OtherEventCallback onUserId;
  OtherEventCallback onSelfEvict;
  List<Map<String, dynamic>> _iceCandidatesParams = <Map<String, dynamic>>[];
  int _idPublishVideo;
  int _idJoinRoom;
  Map<int, String> _idsReceiveVideo = <int, String>{};
  bool _haveRemoteAlreadyInRoom = false;
  List _remoteAlreadyInRoomValues;

  Map<String, dynamic> _iceServers = {
    'iceServers': [
      {
        'url': 'turn:coturn.punkr.app:3478',
        'username': '',
        'credential': ''
      },
    ]
  };

  final Map<String, dynamic> _constraints = {
    'mandatory': {
      'OfferToReceiveAudio': true,
      'OfferToReceiveVideo': true,
    },
    'optional': [],
  };

  Signaling(this._url, this._token, this._session, this._turnUsername, this._turnCredential, this._realUserId, this._realGroupid);

  void updateInternalId() {
    _internalId++;
  }

  close() {
    _localStream?.dispose();
    _localStream = null;
    _timer?.cancel();
    _sendJson(JsonConstants.leaveRoom, null);
    _localPeerConnection?.close();
    _participants.forEach((String id, RemoteParticipant remoteParticipant) {
      remoteParticipant.peerConnection?.close();
    });
    _participants.clear();
    _socket?.close();
  }

  void switchCamera() {
    if (_localStream != null) {
      _localStream.getVideoTracks()[0].switchCamera();
    }
  }

  Future<void> connect() async {
    _iceServers['iceServers'][0]['username'] = _turnUsername;
    _iceServers['iceServers'][0]['credential'] = _turnCredential;
    try {
      _socket = await WebSocket.connect(_url);

      if (this.onStateChange != null) {
        this.onStateChange(SignalingState.ConnectionOpen);
      }

      _socket.listen((data) {
        this.onMessage(json.decode(data));
      }, onDone: () {
        print('Closed by server!');
        if (this.onStateChange != null) {
          this.onStateChange(SignalingState.ConnectionClosed);
        }
      });

      _createLocalPeerConnection().then((_) {
        _idJoinRoom = _sendJson(JsonConstants.joinRoom, {
          JsonConstants.metadata: '{\"clientData\": \"${_realGroupid}_$_realUserId\"}',
          'secret': '',
          'platform': Platform.isAndroid ? 'Android' : 'iOS',
          // 'dataChannels': 'false',
          'session': _session,
          'token': _token,
        });
        _createLocalOffer();
      });

      _startPingTimer();

    } catch(e){
      print('connect error: $e');
      if(this.onStateChange != null){
        this.onStateChange(SignalingState.ConnectionError);
      }
    }
  }

  void onMessage(message) async {
    Map<String, dynamic> mapData = message;
    print('onMessage: $message');
    if (mapData.containsKey(JsonConstants.result)) {
      if (mapData['id'] == _idPublishVideo) {
        _endpointName = mapData[JsonConstants.result]['id'];
      } else if (mapData['id'] == _idJoinRoom) {
        // _createLocalOffer();
      }
      handleResult(mapData[JsonConstants.result], mapData[JsonConstants.id]);
    } else {
      handleMethod(mapData);
    }
  }

  void handleResult(Map<String, dynamic> data, int id) {
    if (data.containsKey(JsonConstants.sdpAnswer)) {
      _saveAnswer(data, id);
    } else if (data.containsKey(JsonConstants.sessionId)) {
      if (data.containsKey(JsonConstants.value)) { // Join room reply
        List<dynamic> values = data[JsonConstants.value];
        _userId = data['id'];
        if (this.onUserId != null){
          this.onUserId(data['id']);
        }
        for (Map<String, dynamic> iceCandidate in _iceCandidatesParams) {
          iceCandidate[JsonConstants.endpointName] = data['endpointName'] ?? data['id'];
          iceCandidate[JsonConstants.id] = data['id'] ?? data['endpointName'];
          _sendJson(JsonConstants.onIceCandidate, iceCandidate);
        }
        if (values.length > 0) {
          _addParticipantsAlreadyInRoom(values);
        }
      }
    } else if (data.containsKey(JsonConstants.value)) {
      print('pong');
    } else {
      print('Unrecognized: $data');
    }
  }

  void handleMethod(Map<String, dynamic> data) {
    if (data.containsKey(JsonConstants.params)) {
      Map<String, dynamic> params = data[JsonConstants.params];
      String method = data[JsonConstants.method];
      switch (method) {
        case JsonConstants.iceCandidate:
          _iceCandidateMethod(params);
          break;
        case JsonConstants.participantJoined:
          _participantJoinedMethod(params);
          break;
        case JsonConstants.sendMessage:
          _messageReceive(params);
          break;
        case JsonConstants.participantPublished:
          _participantPublishedMethod(params);
          break;
        case JsonConstants.participantLeft:
        case JsonConstants.participantEvicted:
          _participantLeftMethod(params);
          break;
        default:
          print('handleMethod: can\'t understand method: $method');
          break;
      }
    } else {
      print('handleMethod, no params');
    }
  }

  Future<void> _addParticipantsAlreadyInRoom(List<dynamic> values) async {
    for (Map<String, dynamic> value in values) {
      String _remoteParticipantId = value[JsonConstants.id];
      RemoteParticipant remoteParticipant = RemoteParticipant(id: _remoteParticipantId, metadata: value[JsonConstants.metadata]);
      if (value.containsKey(JsonConstants.streams)) {
        List<dynamic> streamsList = value[JsonConstants.streams];
        Map<String, dynamic> stream = streamsList?.first;
        if (stream != null && stream.containsKey(JsonConstants.id)) {
          remoteParticipant.streamId = stream[JsonConstants.id];
        }
      }
      _participants[_remoteParticipantId] = remoteParticipant;
      _createRemotePeerConnection(remoteParticipant).then((RTCPeerConnection remotePeerConnection) async {
        _receiveVideoFromParticipant(remoteParticipant);
      });
      if (this.onStateChange != null) {
        this.onStateChange(SignalingState.CallStateConnected);
      }
    }
  }

  void _participantJoinedMethod(Map<String, dynamic> params) {
    RemoteParticipant remoteParticipant = RemoteParticipant(id: params[JsonConstants.id], metadata: params[JsonConstants.metadata]);
    _participants[remoteParticipant.id] = remoteParticipant;
    this.onParticipantsJoined(remoteParticipant);
    _createRemotePeerConnection(remoteParticipant);
    if (this.onStateChange != null) {
      this.onStateChange(SignalingState.CallStateConnected);
    }
  }

  void _messageReceive(Map<String, dynamic> params) {
    print('_message $params');
    if (this.onMessageReceive != null){
      this.onMessageReceive(params);
    }
  }

  void sendMessage(List to, String signalType, dynamic data) {
    Map _msg = {
      'to': to,
      'data': data,
      'type': signalType
    };
    _sendJson(JsonConstants.sendMessage, {
      // 'message': "{\"to\":$to,\"data\":\"$data\",\"type\":\"$signalType\"}"
      'message': jsonEncode(_msg)
    });
  }

  Future<void> _receiveVideoFromParticipant(RemoteParticipant remoteParticipant) async {
    if (remoteParticipant.peerConnection == null) {
      return;
    }
    try {
      RTCSessionDescription sessionDescription = await remoteParticipant.peerConnection.createOffer(_constraints);
      remoteParticipant.peerConnection.setLocalDescription(sessionDescription);
      int id = _sendJson(JsonConstants.receiveVideoFrom, {
        'sender': remoteParticipant.streamId ?? remoteParticipant.id,
        'sdpOffer': sessionDescription.sdp
      });
      _idsReceiveVideo[id] = remoteParticipant.id;
    } catch(e) {
      print(e.toString());
    }
  }

  void _participantLeftMethod(Map<String, dynamic> params) {
    String participantId = params['connectionId'];
    if (participantId == _userId) {
      this.onSelfEvict(_userId);
    }
    else if (_participants.containsKey(participantId)) {
      this.onParticipantsRemove(_participants[participantId]);
      _participants[participantId].peerConnection?.close();
      _participants.remove(participantId);
      if (_participants.length == 0) {
        if (this.onStateChange != null) {
          this.onStateChange(SignalingState.CallStateWaiting);
        }
      }
    }
  }

  void _iceCandidateMethod(Map<String, dynamic> params) {
    bool isLocal = params[JsonConstants.senderConnectionId] == _userId;
    _saveIceCandidate(params, params[JsonConstants.endpointName], params[JsonConstants.senderConnectionId], isLocal);
  }

  _saveIceCandidate(Map<String, dynamic> params, String endpointName, String senderConnectionId, bool isLocal) {
    CustomRTCIceCandidate iceCandidate = CustomRTCIceCandidate(params['candidate'], params['sdpMid'], params['sdpMLineIndex'], endpointName);
    if (isLocal && _localPeerConnection != null) {
      _localPeerConnection.addCandidate(iceCandidate);
    } else if (_participants.containsKey(senderConnectionId)) {
      _participants[senderConnectionId].peerConnection.addCandidate(iceCandidate);
      _participantEndpoints[senderConnectionId] = endpointName;
    }
  }

  void _saveAnswer(Map<String, dynamic> data, int id) {
    RTCSessionDescription sessionDescription = RTCSessionDescription(data['sdpAnswer'], 'answer');
    if (!_localPeerConnectionHasRemoteDescription) {
      _localPeerConnection.setRemoteDescription(sessionDescription);
      _localPeerConnectionHasRemoteDescription = true;
    } else if (_idsReceiveVideo.containsKey(id)) {
      _participants[_idsReceiveVideo[id]].peerConnection.setRemoteDescription(sessionDescription);
    }
  }

  void _startPingTimer() {
    _timer = Timer.periodic(const Duration(seconds: 3), (Timer timer) {
      _sendJson('ping', {
        'interval': '3000'
      });
    });
  }

  Future<MediaStream> createStream({isLocalStream = false}) async {
    final Map<String, dynamic> mediaConstraints = {
      'audio': true,
      'video': {
        'mandatory': {
          'minWidth': '320',
          'minHeight': '240',
          'minFrameRate': '30',
        },
        'facingMode': 'user',
        'optional': [],
      }
    };

    MediaStream stream = await navigator.getUserMedia(mediaConstraints);
    if (isLocalStream && this.onLocalStream != null) {
      this.onLocalStream(stream);
    }
    return stream;
  }

  Future<void> _createLocalPeerConnection() async {
    _localStream = await createStream(isLocalStream: true);
    _localPeerConnection = await createPeerConnection(_iceServers, _constraints);
    _localPeerConnection.onSignalingState = ((state) {
      if (state == RTCSignalingState.RTCSignalingStateStable) {
        
      }
    });
    _localPeerConnection.onIceGatheringState = ((state) {
      if (state == RTCIceGatheringState.RTCIceGatheringStateComplete) {
        if (_haveRemoteAlreadyInRoom) {
          _addParticipantsAlreadyInRoom(_remoteAlreadyInRoomValues);
          _remoteAlreadyInRoomValues.clear();
          _haveRemoteAlreadyInRoom = false;
        }
      }
    });
    _localPeerConnection.onIceCandidate = (candidate) {
      Map<String, dynamic> iceCandidateParams = {
        'sdpMid': candidate.sdpMid,
        'sdpMLineIndex': candidate.sdpMlineIndex,
        'candidate': candidate.candidate,
      };
      if (_userId != null){
        iceCandidateParams[JsonConstants.endpointName] = _endpointName ?? _userId;
        _sendJson(JsonConstants.onIceCandidate, iceCandidateParams);
      } else {
        _iceCandidatesParams.add(iceCandidateParams);
      }
    };
    _localPeerConnection.addStream(_localStream);
  }

  Future<RTCPeerConnection> _createRemotePeerConnection(RemoteParticipant remoteParticipant) async {
    RTCPeerConnection remotePeerConnection = await createPeerConnection(_iceServers, _constraints);
    remotePeerConnection.onIceCandidate = (candidate) {
      Map<String, dynamic> iceCandidateParams = {
        'sdpMid': candidate.sdpMid,
        'sdpMLineIndex': candidate.sdpMlineIndex,
        'endpointName': remoteParticipant.streamId ?? _participantEndpoints[remoteParticipant.id] ?? remoteParticipant.id,
        'candidate': candidate.candidate,
      };
      _sendJson(JsonConstants.onIceCandidate, iceCandidateParams);
    };

    remotePeerConnection.onAddStream = ((stream) {
      remoteParticipant.mediaStream = stream;
      if (this.onParticipantsStreamUpdate != null){
        this.onParticipantsStreamUpdate(remoteParticipant);
      }
      if (this.onAddRemoteStream != null){
        this.onAddRemoteStream(stream);
      }
    });

    remotePeerConnection.onSignalingState = ((state) {
      print('remotePeerConnection.onSignalingState: $state');
      /*
      if (state == RTCSignalingState.RTCSignalingStateStable) {
        
      }
      */
    });

    remotePeerConnection.onIceGatheringState = ((state) {
      print('icestate: $state');
    });

    remotePeerConnection.onRemoveStream = (stream) {
      remoteParticipant.mediaStream = stream;
      if (this.onParticipantsStreamUpdate != null){
        this.onParticipantsStreamUpdate(remoteParticipant);
      }
      if(this.onRemoveRemoteStream != null){
        this.onRemoveRemoteStream(stream);
      }
    };

    remotePeerConnection.onDataChannel = (channel) {
      print('onDataChannel: $channel');
    };

    remoteParticipant.peerConnection = remotePeerConnection;
    return remotePeerConnection;
  }

  Future<void> _createLocalOffer() async {
    try {
      RTCSessionDescription s = await _localPeerConnection.createOffer(_constraints);
      await _localPeerConnection.setLocalDescription(s);
      _idPublishVideo = _sendJson(JsonConstants.publishVideo, {
        'audioOnly': 'false',
        'hasAudio': 'true',
        'doLoopback': 'false',
        'hasVideo': 'true',
        'audioActive': 'true',
        'videoActive': 'true',
        'typeOfVideo': 'CAMERA',
        'frameRate': '30',
        'videoDimensions': '{\"width\": 320, \"height\": 240}',
        'sdpOffer': s.sdp
      });
    } catch (e) {
      print(e.toString());
    }
  }

  Future<void> _participantPublishedMethod(Map<String, dynamic> params) async {
    String _remoteParticipantId = params[JsonConstants.id];
    if (_participants.containsKey(_remoteParticipantId)) {
      RemoteParticipant remoteParticipantPublished = _participants[_remoteParticipantId];
      if (params.containsKey(JsonConstants.streams)) {
        List<dynamic> streamsList = params[JsonConstants.streams];
        Map<String, dynamic> stream = streamsList?.first;
        if (stream != null && stream.containsKey(JsonConstants.id)) {
          remoteParticipantPublished.streamId = stream[JsonConstants.id];
        }
      }
      if (remoteParticipantPublished.peerConnection != null){
        _receiveVideoFromParticipant(remoteParticipantPublished);
      } else {        
        _createRemotePeerConnection(remoteParticipantPublished).then((RTCPeerConnection remotePeerConnection) async {
          _receiveVideoFromParticipant(remoteParticipantPublished);
        });
        
      }
    }
  }

  int _sendJson(String method, Map<String, dynamic> params) {
    Map<String, dynamic> dict = <String, dynamic>{};
    dict[JsonConstants.method] = method;
    dict[JsonConstants.id] = _internalId;
    dict['jsonrpc'] = '2.0';
    if ((params?.length ?? 0) > 0) {
      dict[JsonConstants.params] = params;
    }
    updateInternalId();
    String jsonString = json.encode(dict);
    _socket?.add(jsonString);
    return _internalId - 1;
  }
}