Using a Screen Capture as a Camera Source for Vision

This guide will help you set up your system to send and receive screen capture to Tulip Vision with a USB capture device off-the-shelf or by using free and straightforward tools like Open Broadcaster Studio (OBS) and VLC.

Using a screen capture for real-time visual analytics is a useful tool for monitoring your shop floor in certain situations. The following are some examples for when screen capture can help:

• A visual scanning device (like an x-ray machine for visual inspection) that connects to a computer only works with a proprietary Windows application and cannot be used with a regular camera
• You would like to read text, using OCR, from an application on the screen that may change and help drive your application
• You would like to monitor the state of the computer or any application that may be running on it

By capturing the screen and sending it over the network to Tulip Vision, you can enable analysis of the screen contents with simple means.

Prerequisites

• Tulip Vision is setup and running on your Tulip Instance
• A screen capture USB device (such as this or this) or IP Cameras enabled
• A Windows computer running Tulip Player
• A Windows computer is ready to install applications for screen capture
• If these are not the same computer - both computers need to have an open network connection between them

Using a screen capture USB device

Capture USB devices are a very convenient way to get screen captures into Tulip Vision. We recommend purchasing a regular HDMI-input USB capture device that acts as a camera. There are several options for such a device, such as USB2.0 or UBS3.x, framerate and resolution, which you can select according to your application.

Plug the USB device and connect it to the HDMI output of the PC you are looking to monitor.

The device should now appear as a camera on the Shop Floor page for the Station:

Assign a Camera Configuration to the camera device and observe the output in the configuration page:

You can now use the configuration for Vision detectors and capabilities such as OCR.

If you do not have a USB capture device, the following steps will allow you to capture screen with software only using an IP video stream.

Setting Up Screen Capture on the Windows Machine

Follow these steps to start the screen capture on the designated Windows machine:

1. Download and install OBS version 27.x: https://github.com/obsproject/obs-studio/releases/tag/27.2.4
   Make sure you download versions 27.x and not 28.x since they do not work with the streaming plugin.

2. Download and install OBS RTSP Server plugin: https://github.com/iamscottxu/obs-rtspserver/releases

3. Start OBS and observe the main screen comes up.
   

4. Add either a "Display Capture" or "Window Capture" to the scene.
   

5. Set up the capture to get the view from the application that you wish to monitor. In this example we're going to grab the view from a camera that has a proprietary viewing application.
   

6. You can modify the window capture to crop around just the important part of the application window.
   
   

7. Once you're satisfied with the capture, make sure it covers the entire scene by using the "Fit to Screen" transform.
   
   

8. Use the Tools -> RTSP Server menu to start the RTSP stream. You may keep the default options, unless there are special network circumstances that you know of (like a blocked port).
   
   

Recommended step: Verify the RTSP Stream Works

We highly recommend, before moving on to Tulip Vision, to verify the RTSP stream of the screen capture is indeed working and displaying the correct view. An easy way to do so is to use the free tool VLC Media Player. Make sure to do this on the computer that will receive the RTSP stream - the computer that will be running Tulip Player, if these are not the same computer.

Follow these steps to verify the RTSP stream:

1. Download and install the latest version of VLC: https://www.videolan.org/vlc/

2. Start VLC, observe that it opens successfully without errors.
   

3. Open a new Network Stream from the Media menu.
   

4. In the network media screen enter the URL that corresponds to your stream. In this case we are running the capture on the same computer hence the hostname/address will be localhost. The port is the default 554, and the path is the default from the OBS settings: /live. The full URL will then be `rtsp://localhost:554/live'.
   

5. Observe that you can see the screen capture streamed successfully.
   

Read the Capture Stream on Tulip Vision

The final step is to get the screen capture stream into Tulip Vision, which we will do with an "IP Camera" configuration. This can be easily done via the Tulip UI.

Follow these steps to set up the capture stream on Tulip Vision:

1. Go on Shop Floor -> Vision -> IP Cameras and create a new IP Camera Configuration. Use the URL that you have verified to work on VLC. In this example the capture stream server and client are on the same computer so the URL would be rtsp://localhost:554/live.
   

2. Create a new Camera Configuration for this new "IP Camera" under the "Camera Configurations" tab in Shop Floor -> Vision.
   

3. Assign the new camera configuration to the IP Camera via the Shop Floor page at the station that's configured with the target Tulip Player.
   

4. Allow for the stream to settle and observe the "View" button appear.

5. Click "View" to see the screen capture stream as a Tulip Vision camera.
   

6. At this point the capture stream is just like any other Tulip Vision camera source. You can create detectors and start using the events from Vision to drive your applications.
   

You can now use Tulip Vision to monitor screen captures from a computer, which may sometimes be useful for e.g. monitoring a connected device or a running application. Creating the capture stream is enabled by readily available tools, and configuration of a Tulip Vision "IP Camera".

Further Reading

• Getting Started with Vision
• Using IP Cameras with Vision

Did you find what you were looking for?

You can also head to community.tulip.co to post your question or see if others have faced a similar question!