How do you Fullscreen RTSP Stream in Android LibVLC?

Question

I'm using mrmaffen's VLC-ANDROID-SDK to develop an RTSP streaming app. https://github.com/mrmaffen/vlc-android-sdk

I've had a lot of success getting it working and running quite well, but the problem I'm having that I can't seem to shake is getting it to display the video feed in fullscreen on the SurfaceView, or even just in the center of the SurfaceView.

This is what I get:

http://s1378.photobucket.com/user/Jo_Han_Solo/media/Screenshot_20171214-125504_zps437k1kw2.png.html?filters[user]=146993343&filters[recent]=1&sort=1&o=1

The black window is the total size of the screen, I want that video to fill the screen and hopefully always fill from center, but I can't figure out how to do it.

Anyone have any experience with anything like this and knows how to fix it?

Answer 1

your solution seems interesting, however I'm facing the same issues, which I can't seem to solve (yet) with your approach.

Screenshots of what I got sofar can be seen at: https://photos.app.goo.gl/9nKo22Mkc2SZq4SK9

I also want to (vertically) center an rtsp-video-stream in either landscape/portrait mode on a Samsung-XCover4 (with 720x1280 pixels) and on a device with minimum resolution of 320x480. The minimum Android SDK-version I would love to have it running is API-22 (Android 5.1.1). The libvlc code for which I got the (embedded)VLC-player working, is based on 'de.mrmaffen:libvlc-android:2.1.12@aar'.

Given the above 'requirements', you can see the following behavior in the screenshots. The first two screenshots are on a Samsung-XCover4 (720x1280) where you can see that device-orientation=landscape clips the video and doesn't scale it, whereas the 3rd and 4th screenshot show that the same video-stream doesn't follow the SURFACE_BEST_FIT method (see code below for an explanation) on a device with small-resolution. I would love to see an updateVideoSurfaces to handle the change in device-orientation or at least to show the entire video on startup.

The layout for my VLC-video-player (part of a vertical LinearLayout) is as follows:

<LinearLayout
    android:layout_width="match_parent"
    android:layout_height="0dp"
    android:layout_weight="0.3"
    android:layout_marginBottom="8dp"
    android:layout_marginEnd="8dp"
    android:layout_marginStart="8dp"
    android:layout_marginTop="8dp"
    android:orientation="vertical">

    <FrameLayout
        android:id="@+id/video_surface_frame"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="center"
        android:foregroundGravity="clip_horizontal|clip_vertical"
        tools:ignore="true">

        <ViewStub
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:layout="@layout/surface_view"
            android:id="@+id/surface_stub" />

        <ViewStub
            android:layout_width="1dp"
            android:layout_height="1dp"
            android:layout="@layout/surface_view"
            android:id="@+id/subtitles_surface_stub" />

        <ViewStub
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:layout="@layout/texture_view"
            android:id="@+id/texture_stub" />

    </FrameLayout>

</LinearLayout>

The example code I got from de.mrmaffen uses an updateVideoSurfaces (see below java-code) which uses a number of SURFACE_XX method which to me seem to cover all scenarios with different device-orientations and resolution.

For some reason I can't figure out why this doesn't work and I suspect that the layout I'm using for the player (the FrameLayout/ViewStub's) may cause the issues.

I was wondering if you can shed some light on directions in order to make sure that the video stream will auto-scale/center on any device orientation/resolution.

The player-code I'm using is as follows:

package com.testing.vlc2player;

import ...

public class VLC2PlayerActivity extends AppCompatActivity implements IVLCVout.OnNewVideoLayoutListener,
        IVLCVout.Callback {

    private static final Logger log = LoggerFactory.getLogger(VLC2PlayerActivity.class);

    private static final boolean USE_SURFACE_VIEW = true;
    private static final boolean ENABLE_SUBTITLES = false;
    private static final int SURFACE_BEST_FIT = 0;
    private static final int SURFACE_FIT_SCREEN = 1;
    private static final int SURFACE_FILL = 2;
    private static final int SURFACE_16_9 = 3;
    private static final int SURFACE_4_3 = 4;
    private static final int SURFACE_ORIGINAL = 5;
    private static final int CURRENT_SIZE = SURFACE_BEST_FIT;

    private FrameLayout mVideoSurfaceFrame = null;
    private SurfaceView mVideoSurface = null;
    private SurfaceView mSubtitlesSurface = null;
    private TextureView mVideoTexture = null;
    private View mVideoView = null;

    private final Handler mHandler = new Handler();
    private View.OnLayoutChangeListener mOnLayoutChangeListener = null;

    private LibVLC mLibVLC = null;
    private MediaPlayer mMediaPlayer = null;
    private int mVideoHeight = 0;
    private int mVideoWidth = 0;
    private int mVideoVisibleHeight = 0;
    private int mVideoVisibleWidth = 0;
    private int mVideoSarNum = 0;
    private int mVideoSarDen = 0;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_video_player);

        setupVLCLayout();
    }

    private void setupVLCLayout() {
        log.debug("...");
        final ArrayList<String> args = new ArrayList<>();
        args.add("-vvv");
        mLibVLC = new LibVLC(this, args);
        mMediaPlayer = new MediaPlayer(mLibVLC);

        mVideoSurfaceFrame = findViewById(R.id.video_surface_frame);
        if (USE_SURFACE_VIEW) {
            ViewStub stub = findViewById(R.id.surface_stub);
            mVideoSurface = (SurfaceView) stub.inflate();
            if (ENABLE_SUBTITLES) {
                stub = findViewById(R.id.subtitles_surface_stub);
                mSubtitlesSurface = (SurfaceView) stub.inflate();
                mSubtitlesSurface.setZOrderMediaOverlay(true);
                mSubtitlesSurface.getHolder().setFormat(PixelFormat.TRANSLUCENT);
            }
            mVideoView = mVideoSurface;
        } else {
            ViewStub stub = findViewById(R.id.texture_stub);
            mVideoTexture = (TextureView) stub.inflate();
            mVideoView = mVideoTexture;
        }
    }

    @Override
    protected void onDestroy() {
        super.onDestroy();
        mMediaPlayer.release();
        mLibVLC.release();
    }

    @Override
    protected void onStart() {
        super.onStart();
        final IVLCVout vlcVout = mMediaPlayer.getVLCVout();
        if (mVideoSurface != null) {
            vlcVout.setVideoView(mVideoSurface);
            if (mSubtitlesSurface != null) {
                vlcVout.setSubtitlesView(mSubtitlesSurface);
            }
        } else {
            vlcVout.setVideoView(mVideoTexture);
        }
        vlcVout.attachViews(this);

        String url = getString(R.string.videoURL);
        Uri uri = Uri.parse(url);
        final Media media = new Media(mLibVLC, uri);
        mMediaPlayer.setMedia(media);
        media.release();
        mMediaPlayer.play();

        if (mOnLayoutChangeListener == null) {
            mOnLayoutChangeListener = new View.OnLayoutChangeListener() {
                private final Runnable mRunnable = new Runnable() {
                    @Override
                    public void run() {
                        updateVideoSurfaces();
                    }
                };
                @Override
                public void onLayoutChange(View v, int left, int top, int right,
                                           int bottom, int oldLeft, int oldTop, int oldRight, int oldBottom) {
                    if (left != oldLeft || top != oldTop || right != oldRight || bottom != oldBottom) {
                        mHandler.removeCallbacks(mRunnable);
                        mHandler.post(mRunnable);
                    }
                }
            };
        }
        mVideoSurfaceFrame.addOnLayoutChangeListener(mOnLayoutChangeListener);
    }

    @Override
    protected void onStop() {
        super.onStop();
        if (mOnLayoutChangeListener != null) {
            mVideoSurfaceFrame.removeOnLayoutChangeListener(mOnLayoutChangeListener);
            mOnLayoutChangeListener = null;
        }
        mMediaPlayer.stop();
        mMediaPlayer.getVLCVout().detachViews();
    }

    private void changeMediaPlayerLayout(int displayW, int displayH) {
        log.debug("displayW={}, displayH={}", displayW, displayH);
        /* Change the video placement using the MediaPlayer API */
        int dispWd = displayW;
        int dispHt = displayH;
        dispWd = mVideoSurface.getWidth();  //Note: we do NOT want to use the entire display!
        dispHt = mVideoSurface.getHeight();
        switch (CURRENT_SIZE) {
            case SURFACE_BEST_FIT:
                mMediaPlayer.setAspectRatio(null);
                mMediaPlayer.setScale(0);
                break;
            case SURFACE_FIT_SCREEN:
            case SURFACE_FILL: {
                Media.VideoTrack vtrack = mMediaPlayer.getCurrentVideoTrack();
                if (vtrack == null) {
                    return;
                }
                final boolean videoSwapped = vtrack.orientation == Media.VideoTrack.Orientation.LeftBottom
                        || vtrack.orientation == Media.VideoTrack.Orientation.RightTop;
                if (CURRENT_SIZE == SURFACE_FIT_SCREEN) {
                    int videoW = vtrack.width;
                    int videoH = vtrack.height;
                    if (videoSwapped) {
                        int swap = videoW;
                        videoW = videoH;
                        videoH = swap;
                    }
                    if (vtrack.sarNum != vtrack.sarDen) {
                        videoW = videoW * vtrack.sarNum / vtrack.sarDen;
                    }
                    float ar = videoW / (float) videoH;
                    float dar = dispWd / (float) dispHt;
                    //noinspection unused
                    float scale;
                    if (dar >= ar) {
                        scale = dispWd / (float) videoW; /* horizontal */
                    } else {
                        scale = dispHt / (float) videoH; /* vertical */
                    }
                    log.debug("scale={}", scale);
                    mMediaPlayer.setScale(scale);
                    mMediaPlayer.setAspectRatio(null);
                } else {
                    mMediaPlayer.setScale(0);
                    mMediaPlayer.setAspectRatio(!videoSwapped ? ""+dispWd+":"+dispHt
                            : ""+dispHt+":"+dispWd);
                }
                break;
            }
            case SURFACE_16_9:
                mMediaPlayer.setAspectRatio("16:9");
                mMediaPlayer.setScale(0);
                break;
            case SURFACE_4_3:
                mMediaPlayer.setAspectRatio("4:3");
                mMediaPlayer.setScale(0);
                break;
            case SURFACE_ORIGINAL:
                mMediaPlayer.setAspectRatio(null);
                mMediaPlayer.setScale(1);
                break;
        }
    }

    private void updateVideoSurfaces() {
        log.debug("...");
        int sw = getWindow().getDecorView().getWidth();
        int sh = getWindow().getDecorView().getHeight();
        // sanity check
        if (sw * sh == 0) {
            log.error("Invalid surface size");
            return;
        }

        mMediaPlayer.getVLCVout().setWindowSize(sw, sh);

        ViewGroup.LayoutParams lp = mVideoView.getLayoutParams();
        if (mVideoWidth * mVideoHeight == 0) {
            /* Case of OpenGL vouts: handles the placement of the video using MediaPlayer API */
            lp.width  = ViewGroup.LayoutParams.MATCH_PARENT;
            lp.height = ViewGroup.LayoutParams.MATCH_PARENT;
            mVideoView.setLayoutParams(lp);
            lp = mVideoSurfaceFrame.getLayoutParams();
            lp.width  = ViewGroup.LayoutParams.MATCH_PARENT;
            lp.height = ViewGroup.LayoutParams.MATCH_PARENT;
            mVideoSurfaceFrame.setLayoutParams(lp);
            changeMediaPlayerLayout(sw, sh);
            return;
        }

        if (lp.width == lp.height && lp.width == ViewGroup.LayoutParams.MATCH_PARENT) {
            /* We handle the placement of the video using Android View LayoutParams */
            mMediaPlayer.setAspectRatio(null);
            mMediaPlayer.setScale(0);
        }

        double dw = sw, dh = sh;
        final boolean isPortrait = getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT;

        if (sw > sh && isPortrait || sw < sh && !isPortrait) {
            dw = sh;
            dh = sw;
        }

        // compute the aspect ratio
        double ar, vw;
        if (mVideoSarDen == mVideoSarNum) {
            /* No indication about the density, assuming 1:1 */
            vw = mVideoVisibleWidth;
            ar = (double)mVideoVisibleWidth / (double)mVideoVisibleHeight;
        } else {
            /* Use the specified aspect ratio */
            vw = mVideoVisibleWidth * (double)mVideoSarNum / mVideoSarDen;
            ar = vw / mVideoVisibleHeight;
        }

        // compute the display aspect ratio
        double dar = dw / dh;

        switch (CURRENT_SIZE) {
            case SURFACE_BEST_FIT:
                if (dar < ar) {
                    dh = dw / ar;
                } else {
                    dw = dh * ar;
                }
                break;
            case SURFACE_FIT_SCREEN:
                if (dar >= ar) {
                    dh = dw / ar; /* horizontal */
                } else {
                    dw = dh * ar; /* vertical */
                }
                break;
            case SURFACE_FILL:
                break;
            case SURFACE_16_9:
                ar = 16.0 / 9.0;
                if (dar < ar) {
                    dh = dw / ar;
                } else {
                    dw = dh * ar;
                }
                break;
            case SURFACE_4_3:
                ar = 4.0 / 3.0;
                if (dar < ar) {
                    dh = dw / ar;
                } else {
                    dw = dh * ar;
                }
                break;
            case SURFACE_ORIGINAL:
                dh = mVideoVisibleHeight;
                dw = vw;
                break;
        }

        // set display size
        lp.width  = (int) Math.ceil(dw * mVideoWidth / mVideoVisibleWidth);
        lp.height = (int) Math.ceil(dh * mVideoHeight / mVideoVisibleHeight);
        mVideoView.setLayoutParams(lp);
        if (mSubtitlesSurface != null) {
            mSubtitlesSurface.setLayoutParams(lp);
        }
        // set frame size (crop if necessary)
        lp = mVideoSurfaceFrame.getLayoutParams();
        lp.width = (int) Math.floor(dw);
        lp.height = (int) Math.floor(dh);
        mVideoSurfaceFrame.setLayoutParams(lp);

        mVideoView.invalidate();
        if (mSubtitlesSurface != null) {
            mSubtitlesSurface.invalidate();
        }
    }

    @TargetApi(Build.VERSION_CODES.JELLY_BEAN_MR1)
    @Override
    public void onNewVideoLayout(IVLCVout vlcVout, int width, int height,
                                 int visibleWidth, int visibleHeight,
                                 int sarNum, int sarDen) {
        log.debug("...");
        mVideoWidth = width;
        mVideoHeight = height;
        mVideoVisibleWidth = visibleWidth;
        mVideoVisibleHeight = visibleHeight;
        mVideoSarNum = sarNum;
        mVideoSarDen = sarDen;
        updateVideoSurfaces();
    }

    @Override
    public void onSurfacesCreated(IVLCVout vlcVout) {
        log.debug("vlcVout={}", vlcVout);
    }

    /**
     * This callback is called when surfaces are destroyed.
     */
    public void onSurfacesDestroyed(IVLCVout vlcVout) {
        log.debug("vlcVout={}", vlcVout);
    }

    public void onStopClientMonitoring(View view) {
//        log.info("UI -> Stop monitoring clientId= ...");
//        onBackPressed();
        String androidSDKRelease = Build.VERSION.RELEASE;
        int androidSDKInt = Build.VERSION.SDK_INT;
        String androidInfo = String.format(Locale.getDefault(), "Android %s (Version %d)", androidSDKRelease, androidSDKInt);
        String appVersionName = BuildConfig.VERSION_NAME;
        String appName = getString(R.string.app_name);
        String appInfoTitle = String.format(getString(R.string.app_info_title), appName);
        String infoMsg = String.format(getString(R.string.app_info_message), appVersionName, androidInfo);

        new AlertDialog.Builder(this).setTitle(appInfoTitle)
                .setMessage(infoMsg)
                .setPositiveButton(getString(R.string.button_ok), new DialogInterface.OnClickListener() {
                    @Override
                    public void onClick(DialogInterface dialog, int which) {
                        // Dismiss dialog
                        dialog.dismiss();
                    }
                })
                .create()
                .show();
    }
}

Answer 2

I kind of solved the problem but in a bit of a dodgy way, it's far from complete but considering the lack of knowledge and information on the topic I thought this might help someone for the time being.

1. Find the size of your screen.
2. Set up your final IVLCOut to incorporate the screen size.
3. Adjust setScale to "fullscreen" the video stream.

To explain each task:

1. Setup your globals:

   public class SingleStreamView extends AppCompatActivity implements 
   IVLCVout.Callback {
   
   public int mHeight;
   public int mWidth;
   

Secondly, in the onCreate task find your screen sizes of your device:

    DisplayMetrics displayMetrics = new DisplayMetrics();
        getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);
        mHeight = displayMetrics.heightPixels;
        mWidth = displayMetrics.widthPixels;

2. Then go down to your "CreatePlayer" event and where you set up your video output:

    // Set up video output
        final IVLCVout vout = mMediaPlayer.getVLCVout();
        vout.setVideoView(mSurface);
        vout.setWindowSize(mWidth,mHeight);
        vout.addCallback(this);
        vout.attachViews();

The winning line that made it center in my surface was the "vout.setWindowSize(mWidth,mHeight);"

Then I simply used the setscale option to "fullscreen" the video. That said, it's a bit of a hack way of doing it, and I would like to try and figure out a way to grab the codec information so to dynamically set the scale of the video and that way automatically fullscreen every size video stream to any size screen but for now this will work for known video stream resolutions, it will automatically adjust to the screen size of your phone.

Either way I found that with a Samsung Galaxy s8, a good scaling factor for a 640x480p RTSP stream was 1.8. Coded like so:

        Media m = new Media(libvlc, Uri.parse(RTSP_ADDRESS));
        m.setHWDecoderEnabled(true,false);
        m.addOption(":network-caching=100");
        m.addOption(":clock-jitter=0");
        m.addOption(":clock-synchro=0");
        m.addOption(":fullscreen");
        mMediaPlayer.setMedia(m);
        mMediaPlayer.setAspectRatio("16:9");
        mMediaPlayer.setScale(1.8f);
        mMediaPlayer.play();

Where you got "mMediaPlayer.setScale(1.8f);"

Hope this helps someone!