2. Infrared stream application

2021-08-20
7 min read

Tutorial 2: Infrared stream application

In this tutorial, you will be able to get access to infrared data. Once you run the code, you will have a window displaying the feed of your camera.

Our goal: Every frame that we receive arrives as infrarred data for every pixel of the camera resolution. We need to transform this to a grayscale color range in a dataset. Once converted, we will transfer the data set to create a visible image in our interface.

Prerequisites

  • You must have already followed the guide for installing and configuring the Kinect V2 Link

  • You must have configured your Visual Studio 2019 correctly as instructed Link

  • You already know the general details of opening the camera. Check the Tutorial 01 as this is the template we are following.

Setting up the Infrared feed

Before opening the camera, we need to code and declare the events and variables to retrieve the infrared frames.

Important objects: To handle all the infrared frame data, we will need the following variables:

  • infraredFrameReader: reader for infrared frames
  • infraredFrameData: storage to receive the frame data from the sensor
  • infraredPixels: storage of the converted frames to color pixels for display
  • bitmap: image object in which we transfer the converted frames to show in our XAML interface

Now that we understand our goal and the main variables, let’s start coding the solution:

  1. Include the Kinect library to your source code:
...  
 /// Kinect Libraries
using Microsoft.Kinect;
  1. In the _class MainWindow from your code file, MainWindow.XAML.cs, declare the variables for retrieving the information:
... 
 /// Kinect Sensor
        private KinectSensor kinectSensor = null;
       
        /** INFRARED FRAME**/
        /// Reader for infrared frames
        private InfraredFrameReader infraredFrameReader = null;
        /// Description (width, height, etc) of the infrared frame data
        private FrameDescription infraredFrameDescription = null;
        /// Bitmap to display
        private WriteableBitmap bitmap = null;
  1. Now, to process the incoming infrared frames, we must set up the limits for rendering the infrared data. This will help us translate the information into a readable human eyes image by configuring the intensity values.
...
       /** Scale the values from 0 to 255 **/

        /// Setup the limits (post processing)of the infrared data that we will render.
        /// Increasing or decreasing this value sets a brightness "wall" either closer or further away.
        private const float InfraredOutputValueMinimum = 0.01f;
        private const float InfraredOutputValueMaximum = 1.0f;
        private const float InfraredSourceValueMaximum = ushort.MaxValue;
        /// Scale of Infrared source data
        private const float InfraredSourceScale = 0.75f;
  1. For displaying the data in our interface, we need to create the object Image Source. Once it is called, it will show the values in the _bit map object created in step 1.
...
          // open the sensor
            this.kinectSensor.Open();

            InitializeComponent();
        }

        public ImageSource ImageSource
        {
            get
            {
                return this.bitmap;
            }
  1. Now that we have our objects to store the sensor’s data and show our final results, it is time to process the data. The main logic goes into the MainWindow() function, which currently looks like this:
        public MainWindow()
        {
            // Initialize the sensor
            this.kinectSensor = KinectSensor.GetDefault();

            // open the sensor
            this.kinectSensor.Open();

            InitializeComponent();
        }
  1. To receive the infrared data, we need to open the reader for the depth frames:
...
public MainWindow()
        {
            // Initialize the sensor
            this.kinectSensor = KinectSensor.GetDefault();

            // open the reader for the depth frames
            this.infraredFrameReader = kinectSensor.InfraredFrameSource.OpenReader();

            // wire handler for frame arrival - This is a defined method
            this.infraredFrameReader.FrameArrived += Reader_InfraredFrameArrived;
...

  1. With the reader open, we will start getting one by one the frames of the scene in which the camera is placed. This data will be stored in memory in the Kinect’s buffer. We can read the shape of the data already and verify whether the size of the information is valid to display. If it is correct, then we storage the frame in a data object:
     private void Reader_InfraredFrameArrived(object sender, InfraredFrameArrivedEventArgs e)
        {
           
            // InfraredFrame is IDisposable
            using (InfraredFrame infraredFrame =e.FrameReference.AcquireFrame())
            {
                if (infraredFrame != null)
                {
                    /// We are using WPF (Windows Presentation Foundation)
                    using (Microsoft.Kinect.KinectBuffer infraredBuffer = infraredFrame.LockImageBuffer())
                    {
                        // verify data and write the new infrared frame data to the display bitmap
                        if (((this.infraredFrameDescription.Width * this.infraredFrameDescription.Height) == (infraredBuffer.Size / this.infraredFrameDescription.BytesPerPixel)) &&
                            (this.infraredFrameDescription.Width == this.bitmap.PixelWidth) && (this.infraredFrameDescription.Height == this.bitmap.PixelHeight))
                        {
                            this.ProcessInfraredFrameData(infraredBuffer.UnderlyingBuffer, infraredBuffer.Size);
                        }
                    }
                }
            }

        }
  1. You might notice a warning in the code about the ProcessInfraredFrameData method, which is still undefined. This method allows us to access the image buffer mentioned in the previous step and organize the information inside or bitmap object to be displayed.

First, we lock the image to prevent external access to it. Secondly, we normalize the pixels retrieved to greyscale for each valid frame with the constant values we declared for the intensity. Then, we draw the rectangle shape in which the newly converted pixels exist. Finally, we release the bitmap object for our program to have access to it later.

Note : remember to activate your unsafe code in the Build properties for your project!

  /// Directly accesses the underlying image buffer of the InfraredFrame to create a displayable bitmap.
        /// This function requires the /unsafe compiler option as we make use of direct access to the native memory pointed to by the infraredFrameData pointer.
        /// Activate "unsafe" in the solution properties > on the left >Build > Check Allow unsafe code
        /// <param name="infraredFrameData">Pointer to the InfraredFrame image data</param>
        /// <param name="infraredFrameDataSize">Size of the InfraredFrame image data</param>
private unsafe void ProcessInfraredFrameData(IntPtr infraredFrameData, uint infraredFrameDataSize)
        {
            // infrared frame data is a 16 bit value
            ushort* frameData = (ushort*)infraredFrameData;

            // lock the target bitmap
            this.bitmap.Lock();

            // get the pointer to the bitmap's back buffer
            float* backBuffer = (float*)this.bitmap.BackBuffer;

            // process the infrared data
            for (int i = 0; i < (int)(infraredFrameDataSize / this.infraredFrameDescription.BytesPerPixel); ++i)
            {
                // since we are displaying the image as a normalized grey scale image, we need to convert from
                // the ushort data (as provided by the InfraredFrame) to a value from [InfraredOutputValueMinimum, InfraredOutputValueMaximum]
                backBuffer[i] = Math.Min(InfraredOutputValueMaximum, (((float)frameData[i] / InfraredSourceValueMaximum * InfraredSourceScale) * (1.0f - InfraredOutputValueMinimum)) + InfraredOutputValueMinimum);
            }

            // mark the entire bitmap as needing to be drawn
            this.bitmap.AddDirtyRect(new Int32Rect(0, 0, this.bitmap.PixelWidth, this.bitmap.PixelHeight));

            // unlock the bitmap
            this.bitmap.Unlock();
        }
  1. Finally, we characterize the bitmap object with the data retreived from the InfraredFrameDescription such as Height and Width to create the bitmap object:
...
// wire handler for frame arrival - This is a defined method
            this.infraredFrameReader.FrameArrived += Reader_InfraredFrameArrived;
            // get FrameDescription from InfraredFrameSource
            this.infraredFrameDescription = kinectSensor.InfraredFrameSource.FrameDescription;

            // create the bitmap to display
            this.bitmap = new WriteableBitmap(infraredFrameDescription.Width, infraredFrameDescription.Height, 96.0, 96.0, PixelFormats.Gray32Float, null);

            // Important! Without this we cannot display the image in the interface
            this.DataContext = this;

            // open the sensor
            this.kinectSensor.Open();

            InitializeComponent();
        }
  1. Now, we will configure the interface to display the image we created. Open your interface file by clicking in MainWindow.XAML.cs. You will see a window with the XAML code definition below. XAML is a markup language, so you need to properly close the tags to make it work.

  1. The project already comes with the basic template. We will add elements inside the Grid tag. A grid is composed of rows and columns. So we will define the following tags:
  • Grid.Row Definitions: for controlling the resizing
  • TextBlock: control to display text. We will add the title of our tutorial to inform others what are we displaying
  • ViewBox
  • Image: control to make reference to the ImageSource defined to recall the bitmap object. We use the Binding property as it helps us to interact with data establishing a connection between our source code and our UI (interface).
<Grid Margin="10 0 10 0">
        <Grid.RowDefinitions>
            <RowDefinition Height="Auto" />
            <RowDefinition Height="*" />
            <RowDefinition Height="Auto" />
            <RowDefinition Height="Auto" />
        </Grid.RowDefinitions>
        <TextBlock Grid.Row="0" Margin="0 0 -1 0" HorizontalAlignment="Left" VerticalAlignment="Bottom" FontFamily="Segoe UI" FontSize="18">Tutorial 02: Infrared view</TextBlock>
        <Viewbox Grid.Row="1" HorizontalAlignment="Center">
            <Image Source="{Binding ImageSource}" Stretch="UniformToFill" />
        </Viewbox>
    </Grid>
  1. Save everything. Now, Build and Run your code. Click on the green play button to start. You should get a result like this:

  1. Congratulations! You finished Tutorial 02. Check the complete source code in the repository for a complete overview Link

Questions?

If you have any questions, comments, or suggestions, feel free to contact me on Twitter or Linkedin as @violetasdev.

References

Avatar
Violeta Sosa León Microservices Integration, DevOps, Software Development, Geospatial Technologies, Spatial Data Science, Software Development Teams Management