DUIXSDKAndroid working with documents

# DUIX_SDK_Android working with documents

updated: 2022-09-14 14:04:13

This article describes how to use the DUIX Android SDK provided by Silicon Digital People Services, including downloads and installations, key interfaces, and code samples.

# material preparation

duix_sdk_release_1.3.9.aar

# download and install

download duix_sample_android_1.1.32.zip (opens new window)
Unzip the zip package, get the SDK package in AAR format in the duix_sample/app/libs directory, and integrate the AAR package into your project to rely on.
Open this project using Android Studio to view the reference code implementation, where the main code for the duix interaction is in the DUIXActivity .java file.

# Gradle configuration

add the following configurations to the app module under the project:

dependencies {
    implementation "org.igniterealtime.smack:smack-android-extensions:4.3.0"
    implementation "org.igniterealtime.smack:smack-tcp:4.3.0"
    implementation 'org.webrtc:google-webrtc:1.0.32006'
    
    ...
}

configurations {
    all {
        exclude group: 'xpp3', module: 'xpp3'
    }
}

# permission processing

In the SDK, manifest has added network, audio, Bluetooth and other permission applications, if you need to turn on the camera need to add camera permissions

<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />
<uses-feature
        android:glEsVersion="0x00020000"
        android:required="true" />

<uses-permission android:name="android.permission.CAMERA" />

# SDK KEY INTERFACES

Use DUIXFactory.getDUIX() to get a DUIXCore instance

/**
* @param listener: DUIX回调接口
*/
public void addDUIXListener(IDUIXListener listener)

public removeDUIXListener(IDUIXListener listener)

The IDUIXListener interface contains the following callbacks:

onIMConnected: The IM module connects successfully, and when the connect function is called in the connected state, it will also trigger a callback.
```
default void onIMConnected()
```
onIMConnectError: Im connection fails when im is triggered by an abnormal interruption and proxy user drop.
```
/**
* @param msg  wrong message description
*/
default void onIMConnectError(String msg)
```

onRender: Returns the background digital person node information

/**
 * Call back the node information for the background execution task
 *
 * @param id    Render node ID
 * @param name  Render node ID name
 */
default void onRender(String id, String name)

onRenderState: Returns the background rendering state

/**
 * Returns the background rendering state. A progress bar can be drawn based on the Loading value. Show indicates that the background has completed the work of loading the stream, and you can hide the loading box at this time to send a client ready signal to start the call process.
 * You can obtain the digital human voice broadcast status in this callback(playStart、playStop)
 * @param state Loading、Ready、Show、playStart、playStop
 * @param dataJson  The complete structure of the data
 */
default void onRenderState(String state, String dataJson) {
}

onDetectedSpeech: Returns the recognition result

/**
 * The recognition result is returned
 *
 * @param asrText ASR  Identified content
 */
default void onDetectedSpeech(String asrText) {
}

onBusy: The background server is busy

/**
 *The background resources are busy and cannot be accessed
 */
default void onBusy() {
}

available: background heartbeat, default 5s execution once.

/**
 * Active messages are sent regularly, and the default 5s is executed once. You can send a session pingSession() task in this callback
 */
default void available() {
}

onCommand: Returns session node information

/**
 * Returns the operation information of the digital person in the background
 *
 * @param type     The type of operation
 * @param flowId   Do persistence with nodeId, so that the next time you come in makeSession, you can directly restore to the current node
 * @param nodeId   Do persistence with flowId, so that the next time you come in makession, you can directly restore to the current node
 * @param json   The entire message content
 */
default void onCommand(String type, String flowId, String nodeId, String json) {
}

onByeBye: Remote end message

/**
 * The remote end message
 * @param uuid      Filter through this UUID whether it is your own session. The ID of other sessions that may be returned in takeover mode。
 * @param code      An error code is returned for reasons such as authentication errors
 * @param reason    Error messages are returned for reasons such as authentication errors
 */
default void onByeBye(String uuid, int code, String reason){
}

onRTCError: The SDP setting exception that occurred when creating an RTC session
```
default void onRTCError(String message) {
}
```

onIceConnectionChange: RTC connection status

/**
* rtc    Connection status callbacks
* @param iceConnectionState     The current rtc status
*/
default void onIceConnectionChange(PeerConnection.IceConnectionState iceConnectionState) {
    }

status list:

name	illustrate
NEW	THE ICE PROXY IS COLLECTING ADDRESSES OR WAITING FOR A REMOTE CANDIDATE TO BE AVAILABLE.
CHECKING	THE ICE AGENT HAS RECEIVED AT LEAST ONE REMOTE CANDIDATE AND IS VALIDATED REGARDLESS OF WHETHER A CONNECTION IS AVAILABLE AT THIS TIME. CANDIDATES MAY ALSO CONTINUE TO BE COLLECTED.
CONNECTED	THE ICE AGENT HAS FOUND AT LEAST ONE AVAILABLE CONNECTION FOR EACH CANDIDATE, AT WHICH POINT IT WILL CONTINUE TO TEST THE REMOTE CANDIDATE TO FIND A BETTER CONNECTION. CANDIDATES MAY ALSO CONTINUE TO BE COLLECTED.
COMPLETED	THE ICE AGENT HAS DISCOVERED AVAILABLE CONNECTIONS AND IS NO LONGER TESTING REMOTE CANDIDATES.
FAILED	ICE CANDIDATES TESTED ALL REMOTE CANDIDATES AND FOUND NO MATCHING CANDIDATES. IT IS ALSO POSSIBLE THAT SOME OF THE AVAILABLE CONNECTIONS WERE FOUND IN SOME CANDIDATES.
DISCONNECTED	the test is no longer active, which may be a temporary state that can self-recover.
CLOSED	THE ICE PROXY IS TURNED OFF AND NO LONGER ANSWERS ANY REQUESTS.

onAddTrack: Remote media stream callback

/**
 * UNIFIED_PLAN    Use this callback to get the remote media stream
 *
 * @param track     Media object base class。    could be VideoTrack和AudioTrack
 */
default void onAddTrack(MediaStreamTrack track, MediaStream[] mediaStreams) {
}

onAddStream: Remote media stream callback, which uses the onAddTrack callback of the UNIFIED_PLAN protocol by default.

/**
 * PLAN_B    Mode uses changed object encapsulationVideoTrack和AudioTrack
 *
 * @param mediaStream     The streaming media object encapsulation
 */
default void onAddStream(MediaStream mediaStream) {
}

onLocalAudioSamples: Local audio data callback that requires setEnableAudioSamples(true) when setting RTCOptions when initializing a session

/**
 * Local audio data callbacks that require RTCOptions to setEnableAudioSamples when initializing a session(true)
 * @param audioSamples
 */
default void onLocalAudioSamples(JavaAudioDeviceModule.AudioSamples audioSamples) {
}

onRemoteAudioSamples: Remote audio data callback that requires setEnableAudioSamples(true) when setting RTCOptions when initializing a session

/**
 * Remote audio data callbacks that require setEnableAudioSamples when RTCOptions is initialized(true)
 * @param audioSamples
 */
default void onRemoteAudioSamples(JavaAudioDeviceModule.AudioSamples audioSamples) {
}

onCameraSwitch: Switch cameras during interaction

/**
 * Callback for the result of the camera switchover
 *
 * @param success     Whether the switchover was successful
 * @param back        Whether it is the rear camera
 * @param msg         If the switch fails, the reason for the failure is returned
 */
default void onCameraSwitch(boolean success, boolean back, String msg) {
}

onStartVideoRecord: The video starts recording

/**
 * Video starts recording (with audio)
 */
default void onStartVideoRecord() {
}

onStopVideoRecord: End of video recording

/**
 *The video recording ends
 * @param path     The path where the video is saved
 */
default void onStopVideoRecord(String path) {
}

onPauseVideoRecord, onResumeVideoRecord

// Video pause and resume
default void onPauseVideoRecord() {
}
default void onResumeVideoRecord() {
}

onVideoRecordTime: Returns the video recording time

/**
 * Returns the time of recording
 *
 * @param timeStampAudio     The length of time the recording, unit ms /1000/1000=s
 * @param nanoTime       Current time, in nanosecond units
 * @param baseTimeStamp    start time
 * @param pauseDelay     pause time
 */
default void onVideoRecordTime(long timeStampAudio, long nanoTime, long baseTimeStamp, long pauseDelay) {
}

onAllocate: Seat assignment notification

/**
 * When the client initiates a session, the assignment option is turned on, and agents in the same organization receive the message
 * @param uuid      Client sessionID
 */
default void onAllocate(String uuid) {
}
default void onAllocate(String uuid) {
}

onTakeOverConnected: Monitors client sessions.
onTakeOverStart Success: Started regulation successfully
onTakeOverStopSuccess: End of takeover success

onAnswer: Synchronizes the client reply to the takeover side

/**
 * Synchronize the user-side reply to the takeover side
 * @param uuid    User sessions
 * @param content     Session content
 */
default void onAnswer(String uuid, String content) {
}

onAssistantChatCreated: The secondary session was created successfully

// A secondary session can do some of the need for direct communication between two clients
default void onAssistantChatCreated(String target) {
}

onSupply Meshage: Supplemented by an auxiliary session protocol

connect: Connect to the IM server

 /**
     * Connect to the IM server.
     * @param imOptions IMConnection parameters
     */
public void connect(Context context, DUIXFactory.IMOptions imOptions)

disconnect: Disconnects the IM connection

// Disconnecting the IM connection generally does not need to be called, and in the scenario where the IM needs to accompany the user login throughout the life cycle, the connection of the message notification needs to be disconnected by this method.
public void disconnect()

isOnline: Check if im is incumbentally connected

createSession: Creates a session

/**
     * Create a session
     * @param context contesxt
     * @param eglBaseContext and Surface Shared gl context
     * @param sessionOptions     Session configuration, which defines the ID, resource group, docking mode, and other information of the session
     * @param rtcOptions rtc     Docking parameters, which define the media parameters
     */
public void createSession(Context context, EglBase.Context eglBaseContext, DUIXFactory.SessionOptions sessionOptions, DUIXFactory.RTCOptions rtcOptions)

reCreateSession: Re-create the session

sendByBot: Sends a text message in bot mode

/**
* bot   Send text messages in bot mode
* @param uuid   session id
* @param text   Text that needs to be sent to the background
*/
public void sendByBot(String uuid, String text)

sendByClient: Sends messages in client-driven mode

/**
* The message sent in client-driven mode can be text, it can be the url of wav, and when sending wavUrl, the text field is placed with a "" string
* @param uuid   session id
* @param text   Text that needs to be sent to the background
* @param wavUrl   Text that needs to be sent to the background
*/
public void sendByClient(String uuid, String text, String wavUrl)

clientReady: Notifies the background to start playing the phone

ASRSwitch: Asr switch in long connection recognition mode

/**
* Long connection recognition mode asr switch
* @param uuid    session id
* @param asrSwitch    Notifications background to turn asr on or off
*/
public void asrSwitch(String uuid, boolean asrSwitch)

switchCamera: Switch cameras when the front and rear cameras are supported

setMicrophone: Sets whether the microphone is muted

/**
* Sets whether the microphone is muted
* @param enable  true: Transmit sound，false:mute
*/
public boolean setMicrophone(boolean enable)

audioBreak: Audio playback interrupt

/**
* Audio playback interrupt
*/
public void audioBreak(String UUID)

pingSession: The session pings the message, keeping the conversation connection active
```
/**
* Send regularly to keep the connection active
*/
public void pingSession()
```

closeSession: Closes the session

/**
* shut session
* @param uuid session id
*/
public void closeSession(String uuid)

release: Releases the resource and is called in Activity's onDestroy

/**
* Turn off RTC communication while closing the IM connection, freeing up resources.
*/
public void release()

releaseSession: Frees the session, which is called in Activity's onDestroy

/**
* When it is necessary to maintain the IM communication connection state is to call the change function to release the RTC resources.
*Turn off RTC communication but keep the IM connection
*/
public void releaseSession()

createTakeOver: Create a takeover session monitoring in an agent scenario, with createSession.
takeOverStart: Initiates a takeover in an agent scenario
takeOverStop: End of takeover in the agent scene

# invoke the step
1. Use DUIXFactory.getDUIX() to obtain a DUIXCore instance.
2. Use mDUIXCore.connect(imOptions) to establish an IM connection.
3. After the connection is successful, mDUIXCore.createSession() initiates a session
4. Gets the media stream in the onAddTrack callback and binds to the renderer
5. Available callbacks are sent in a regular pingSession persistent session.
6. Interact with the background in different ways depending on the boot mode (sendByClient, sendByBot)
7. close the session and use release to release resources

# Proguard configuration

if the code uses obfuscation, configure it in the proguard-rules.pro:

-keep class org.webrtc.**{ *; }
-keep class org.igniterealtime.**{ *; }
-keep class org.jivesoftware.**{ *; }
-keep class org.jxmpp.**{ *; }
-keep class org.jivesoftware.**{ *; }

# code example

protected void permissionOk() {
    mDUIXCore = DUIXFactory.getDUIX();
    mDUIXCore.addDUIXListener(mDUIXCallback);
}

IM PARAMETER SETTING AND CONNECTION

String deviceId = "endpoint_android_" + DeviceUtils.getUUID(mContext);
DUIXFactory.IMOptions imOptions = new DUIXFactory.IMOptions(deviceId)   // 客户端ID
                .setImHost(mConfig.getHost())
                .setImPort(mConfig.getPort())
                .setImUserJID(mConfig.getJID());
mDUIXCore.connect(mContext, imOptions);                                                                                      // Start the connection

@Override
public void onIMConnected() {
      // The connection is successful and the session is initiated
}

SET THE SESSION PARAMETERS AND RTC MEDIA INFORMATION TO START THE SESSION

public void onIMConnected() {
            if (mDUIXCore != null) {
            UUID = IMMessageFormat.getRandomUUID();
            // Session parameters
            DUIXFactory.SessionOptions sessionOptions = new DUIXFactory.SessionOptions(UUID, mConfig.getToken(), mConfig.getAppId())
                    .setRobotMode(mConfig.getRobotMode())         // bot mode:  "bot" or ""
                    .setGroupId(mConfig.getGroupId()) //  Resource group

                    .setFlowId(mLastSessionFlowId)    // Resume to the session
                    .setNodeId(mLastSessionNodeId)    // Resume to the session
                    .setAsrEnable(mConfig.isAsr());
             // RTC media parameters
            DUIXFactory.RTCOptions rtcOptions = new DUIXFactory.RTCOptions()
                    .setAudioEnable(mConfig.getAsrMode() == 1)     // default asrMode==1Long speech recognition，The results are returned in the background。asrMode=0  You need to connect to the three-party identification interface by yourself.
                    .setVideoEnable(mConfig.isCamera());           // Whether to transfer the video
            mDUIXCore.createSession(mContext, eglBaseContext, sessionOptions, rtcOptions);
        }
}

@Override
protected void renderState(String state, String dataJson){
  // When you receive a change message, the background load is complete
  if ("Show".equals(state)) {
      // The sending client is ready to complete the signal and start playing the words
      mDUIXCore.clientReady(UUID);
      // UI   Hides the load box
  }
}

send messages to interact with the background

// Depending on the robotMode, decide which interface to use to send the message
private void sendIMMsg(String msg) {
        if ("bot".equals(mConfig.getRobotMode())) {
            // bot Mode passes text directly
            mDUIXCore.sendByBot(UUID, msg);
        } else {
            mDUIXCore.sendByClient(UUID, msg, "");
        }
}

@Override
public void wavClick(String url) {
      if (mDUIXCore != null) {
            mDUIXCore.sendByClient(UUID, url);
            appendMsg("send wav url: " + url);
      }
}

the video preview control is initialized

//RTC modules and controls share that context
eglBaseContext = EglBase.create().getEglBaseContext();
remote_renderer.init(eglBaseContext, null);
remote_renderer.setScalingType(RendererCommon.ScalingType.SCALE_ASPECT_FILL);
remote_renderer.setMirror(false);                       // Use this tag to mirror the picture
remote_renderer.setEnableHardwareScaler(false /* enabled */);

local_renderer.init(eglBaseContext, null);
local_renderer.setScalingType(RendererCommon.ScalingType.SCALE_ASPECT_FILL);
local_renderer.setMirror(true);
local_renderer.setEnableHardwareScaler(false /* enabled */);
local_renderer.setZOrderOnTop(true);                        // Use this tag to mirror the picture
local_renderer.setZOrderMediaOverlay(true);         // Handle display issues caused by multiple surfaceview overrides
local_renderer.setOnTouchListener(
    new RendererTouchListener(
        new Rect(0, 0, DisplayUtils.getScreenWidth(mContext), DisplayUtils.getScreenHeight(mContext))));    // Add custom touch events to control where controls appear

video signal docking

@Override
public void onAddTrack(MediaStreamTrack track, MediaStream[] mediaStreams) {
      if (track instanceof VideoTrack) {
          // Local video footage is also displayed
          VideoTrack localVideo = mDUIXCore.getLocalVideoTrack();
          if (localVideo != null) {
                localVideo.addSink(small_renderer);
          }
          //  Demonstrate the remote signal
          VideoTrack remoteVideoTrack = (VideoTrack) track;
          remoteVideoTrack.addSink(fullscreen_renderer);
      }
}

close the session and release the resources

@Override
protected void onDestroy() {
        super.onDestroy();
        if (fullscreen_renderer != null) {
            fullscreen_renderer.release();
        }
        if (small_renderer != null) {
            small_renderer.release();
        }
        if (mDUIXCore != null) {
            if (UUID != null) {
                mDUIXCore.closeSession(UUID);
            }
            mDUIXCore.removeDUIXListener(mDUIXCallback);
            mDUIXCore.release();
        }
}

# precautions

the rtc module requires a microphone, possibly a camera. requires a dynamic request for permissions.
After some time the session automatically closes to see if a pingSession() signal is sent in the avatar() callback.