3 the link from "Please refer to this blog regarding" passage was missing, the entire section is an unformatted copy without proper quoting and where it came from, replacing it with the link to the correct page
source | link

FreeRDP: A Remote Desktop Protocol Implementation

FreeRDP is a free implementation of the Remote Desktop Protocol (RDP), released under the Apache license. Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience.

Resources

Project website: http://www.freerdp.com/
Issue tracker: https://github.com/FreeRDP/FreeRDP/issues
Sources: https://github.com/FreeRDP/FreeRDP/
Wiki: https://github.com/FreeRDP/FreeRDP/wiki
Downloads and other resources: http://pub.freerdp.com
API doc: http://pub.freerdp.com/api/

IRC channel: #freerdp @ irc.freenode.net
Mailing list: https://lists.sourceforge.net/lists/listinfo/freerdp-devel

Microsoft Open Specifications

Information regarding the Microsoft Open Specifications can be found at: http://www.microsoft.com/openspecifications/

A list of reference documentation is maintained here: https://github.com/FreeRDP/FreeRDP/wiki/Reference-Documentation

Compilation

Instructions on how to get started compiling FreeRDP can be found on the wiki: https://github.com/FreeRDP/FreeRDP/wiki/Compilation

Multimedia Redirection According to [MS-RDPEV], Multimedia Redirection is “used to transfer synchronized audio and video data from a terminal server to a terminal client. The client can play the audio and video data and synchronize this data by using the timing information provided by this protocol”. It’s also called “Video Redirection”.

Usage

Server Requirement

The Multimedia Redirection requires the server running the following Windows version:

Windows 2008 R2: audio and video playback redirection must be enabled manually. Windows 7 Ultimate/Enterprise: works out of the box. Please refer to this blog regarding how to configure Windows 2008 R2.

Note that currently only Windows Media Player supports this feature.

Compilation

The following development packages must be installed to have full Multimedia Redirection features compiled in.

FFmpeg (libavcodec-dev) ALSA (libasound2-dev) and/or PulseAudio (libpulse-dev) XVideo (libxv-dev) Quick Start

To use the feature, register the “tsmf” dynamic virtual channel as following:

xfreerdp --plugin drdynvc --data tsmf -- (server)

Audio Device

By default, FreeRDP will detect PulseAudio availability and then falls back to ALSA, and then use the default audio device. You can force an audio backend and optionally a sound device by adding additional arguments after tsmf. The following example forces to use the default audio device with PulseAudio:

xfreerdp --plugin drdynvc --data tsmf:audio:pulse -- (server)

The following example forces to use the audio device “plughw:0,0” with ALSA:

xfreerdp --plugin drdynvc --data tsmf:audio:alsa:plughw:0,0 -- (server)

Note that you do not need to register the rdpsnd plugin to have audio playback. The tsmf plugin generally gives higher audio quality than rdpsnd.

XVideo Adaptor

By default, FreeRDP will choose the base port of the last available XVideo adaptor, which should just work in most cases. However sometimes you might want to manually choose a different port, for example, the default adaptor does not work well, or the port is being used by other process. You can specify the —xv-port argument:

xfreerdp --xv-port (port) --plugin drdynvc --data tsmf -- (server)

For a complete list of all available adaptors and ports in your system, run “xvinfo”.

Demuxing and Decoding

Every video file is basically constructed in two steps. Firstly, raw video and audio streams are encoded using some kind of algorithms, called “codec”, such as H264, WVC1, MP3, AC3, etc. This step is called “Encoding”. Then the encoded streams are packed together into a single file using specific container format, such as Matroska(mkv), AVI, MP4, etc. This step is called “Muxing”.

So in order to play a video file, two steps called “Demuxing” and “Decoding” are performed respetively. Those are done transparently when the video is played locally. However when we usewith Multimedia Redirection, it’s very important to understand that those two steps are performed on different host and different stage: demuxing is performed by the server first, then decoding is performed by the client afterwards. When a video file is opened on the server, the media player will first parse the video file container, retrieve all necessary header information, and separate the content of all video and audio streams. Then the server sends them to the client. The client needs to prepare necessary codec to decode the streams and play them.

It’s obvious that in order to be able to redirect specific video file, the server MUST understand the container format. For example, by default Window Media Player does not support Matroska. You must install the Matroska filter on the server to play mkv files. However, the server does not need to have the codec installed, since once demuxing is done, the original stream content are sent to the client for decoding.

Supported Codecs

Currently the following codecs are supported and tested.

Video:

WVC1 WMV3 H264/AVC1 Audio:

WMA2 MP3 AAC AC3 Bandwidth

Based on some observations, 100M ethernet is pretty much enough play HD 720P content as if it plays locally. When testing with 54M Wifi (which has much higher latency), a DVD quality video can still play smoothly, but sometimes there will be some noticeable frame drops.

Test Scenario

Here is a test scenario that is known to work:

Download Big Buck Bunny in 854×480 resolution and H.264 format. By default, Windows Server 2008 R2 SP1 won’t be able to play the video. Install K-Lite Codec Pack Basic to get the proper codecs.

Once you know that the server is able to play the video, try connecting and playing the video remotely. If it works, you should get something smooth enough like this:

Hacking

Adding a New Codec

A stream has three main attributes in order to be understood by the client: Major Type, Sub Type and Format Type.

Major Type

There are currently only two major types: video and audio. They are defined as fixed GUID, and mapped to constants defined by our own, in tsmf_constants.h. There could be the third useful major type – subtitle, however it seems that Windows Media Player is not redirecting it even though the subtitle stream is available in the video file.

Sub Type

Sub types are actually the codec needed to decode the stream. Again they are defined as fixed GUID, and we map them to constants in tsmf_constants.h. When the sub types are passed to FFmpeg, the codec constants are mapped again into the FFmpeg codec enum. It’s necessary to separate the mapping into two steps and have our own abstraction layer, since FFmpeg is designed as a sub-plugin and we could support other decoding library in the future.

Format Type

Codecs require information to be able to start decoding properly. For example, the dimension of the video size. Many codecs also require certain codec-specific extra data. In Matroska specification, such extra data is called “CodecPrivate”. In RDP protocol, all those information is packed into a struct called Format Type, defined mostly in MF (Media Foundation) or DirectShow API. Every Format Type is given a fixed GUID and again we map them to our own constants.

Sometimes it could become tricky to retrieve the codec extra data from the Format Type struct. In this case, the Matroska container format might become handy, since it’s much easier to be analyzed by some EBML tree viewer application.

Pixel Format

After a video frame in a stream is decoded, the outcome is a raw image consists of width x height pixels. Those pixels are packed in certain pixel format, usually YUV. Then the image is passed to XVideo directly for hardware decoding and scaling.

The video codecs we currently supported are all producing I420, which is probably the most common format. We also support converting I420 to YV12 in case the video driver does not support it. However, it is yet to investigate whether other pixel formats exist.

Known Issues

The “Landscapes.wtv” file shipped with Windows 7 by default won’t be able to play. It’s difficult to analyze this file format, since FFmpeg does not even able to convert it to mkv. Fast-forwarding is not yet implemented. Volume control is not yet implemented. FAQ

FreeRDP: A Remote Desktop Protocol Implementation

FreeRDP is a free implementation of the Remote Desktop Protocol (RDP), released under the Apache license. Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience.

Resources

Project website: http://www.freerdp.com/
Issue tracker: https://github.com/FreeRDP/FreeRDP/issues
Sources: https://github.com/FreeRDP/FreeRDP/
Wiki: https://github.com/FreeRDP/FreeRDP/wiki
Downloads and other resources: http://pub.freerdp.com
API doc: http://pub.freerdp.com/api/

IRC channel: #freerdp @ irc.freenode.net Mailing list: https://lists.sourceforge.net/lists/listinfo/freerdp-devel

Microsoft Open Specifications

Information regarding the Microsoft Open Specifications can be found at: http://www.microsoft.com/openspecifications/

A list of reference documentation is maintained here: https://github.com/FreeRDP/FreeRDP/wiki/Reference-Documentation

Compilation

Instructions on how to get started compiling FreeRDP can be found on the wiki: https://github.com/FreeRDP/FreeRDP/wiki/Compilation

Q: What does “TSMF” mean? A: “TSMF” is the channel name of the Multimedia Redirection protocol, but unfortunately there’s no official explanation in the MS documentation. A reasonable guess would be “Terminal Service Media Foundation”.https://github.com/FreeRDP/FreeRDP/blob/master/README

Q: I have enough bandwidth and I have registered the tsmf plugin. However my video is very slow like slideshow. A: The video codec is not yet supported, so the video frames are decoded on the server and sent to the client as normal RDP bitmaps. Please look at the standard output for some useful information.

Multimedia Redirection

Q: I have registered the tsmf plugin but I cannot hear audio. However if I also registered rdpsnd I can hear it. A: The audio codec is not yet supported, so the audio samples are decoded on the server and sent to the client through rdpsnd plugin. If rdpsnd is not registered, you will not be able to hear anything. But note that even you can hear sound through rdpsnd, video/audio will be out of sync. So the best solution is to request the audio codec support.https://github.com/FreeRDP/FreeRDP/wiki/Multimedia-Redirection

FreeRDP: A Remote Desktop Protocol Implementation

FreeRDP is a free implementation of the Remote Desktop Protocol (RDP), released under the Apache license. Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience.

Resources

Project website: http://www.freerdp.com/
Issue tracker: https://github.com/FreeRDP/FreeRDP/issues
Sources: https://github.com/FreeRDP/FreeRDP/
Wiki: https://github.com/FreeRDP/FreeRDP/wiki
Downloads and other resources: http://pub.freerdp.com
API doc: http://pub.freerdp.com/api/

IRC channel: #freerdp @ irc.freenode.net
Mailing list: https://lists.sourceforge.net/lists/listinfo/freerdp-devel

Microsoft Open Specifications

Information regarding the Microsoft Open Specifications can be found at: http://www.microsoft.com/openspecifications/

A list of reference documentation is maintained here: https://github.com/FreeRDP/FreeRDP/wiki/Reference-Documentation

Compilation

Instructions on how to get started compiling FreeRDP can be found on the wiki: https://github.com/FreeRDP/FreeRDP/wiki/Compilation

Multimedia Redirection According to [MS-RDPEV], Multimedia Redirection is “used to transfer synchronized audio and video data from a terminal server to a terminal client. The client can play the audio and video data and synchronize this data by using the timing information provided by this protocol”. It’s also called “Video Redirection”.

Usage

Server Requirement

The Multimedia Redirection requires the server running the following Windows version:

Windows 2008 R2: audio and video playback redirection must be enabled manually. Windows 7 Ultimate/Enterprise: works out of the box. Please refer to this blog regarding how to configure Windows 2008 R2.

Note that currently only Windows Media Player supports this feature.

Compilation

The following development packages must be installed to have full Multimedia Redirection features compiled in.

FFmpeg (libavcodec-dev) ALSA (libasound2-dev) and/or PulseAudio (libpulse-dev) XVideo (libxv-dev) Quick Start

To use the feature, register the “tsmf” dynamic virtual channel as following:

xfreerdp --plugin drdynvc --data tsmf -- (server)

Audio Device

By default, FreeRDP will detect PulseAudio availability and then falls back to ALSA, and then use the default audio device. You can force an audio backend and optionally a sound device by adding additional arguments after tsmf. The following example forces to use the default audio device with PulseAudio:

xfreerdp --plugin drdynvc --data tsmf:audio:pulse -- (server)

The following example forces to use the audio device “plughw:0,0” with ALSA:

xfreerdp --plugin drdynvc --data tsmf:audio:alsa:plughw:0,0 -- (server)

Note that you do not need to register the rdpsnd plugin to have audio playback. The tsmf plugin generally gives higher audio quality than rdpsnd.

XVideo Adaptor

By default, FreeRDP will choose the base port of the last available XVideo adaptor, which should just work in most cases. However sometimes you might want to manually choose a different port, for example, the default adaptor does not work well, or the port is being used by other process. You can specify the —xv-port argument:

xfreerdp --xv-port (port) --plugin drdynvc --data tsmf -- (server)

For a complete list of all available adaptors and ports in your system, run “xvinfo”.

Demuxing and Decoding

Every video file is basically constructed in two steps. Firstly, raw video and audio streams are encoded using some kind of algorithms, called “codec”, such as H264, WVC1, MP3, AC3, etc. This step is called “Encoding”. Then the encoded streams are packed together into a single file using specific container format, such as Matroska(mkv), AVI, MP4, etc. This step is called “Muxing”.

So in order to play a video file, two steps called “Demuxing” and “Decoding” are performed respetively. Those are done transparently when the video is played locally. However when we use Multimedia Redirection, it’s very important to understand that those two steps are performed on different host and different stage: demuxing is performed by the server first, then decoding is performed by the client afterwards. When a video file is opened on the server, the media player will first parse the video file container, retrieve all necessary header information, and separate the content of all video and audio streams. Then the server sends them to the client. The client needs to prepare necessary codec to decode the streams and play them.

It’s obvious that in order to be able to redirect specific video file, the server MUST understand the container format. For example, by default Window Media Player does not support Matroska. You must install the Matroska filter on the server to play mkv files. However, the server does not need to have the codec installed, since once demuxing is done, the original stream content are sent to the client for decoding.

Supported Codecs

Currently the following codecs are supported and tested.

Video:

WVC1 WMV3 H264/AVC1 Audio:

WMA2 MP3 AAC AC3 Bandwidth

Based on some observations, 100M ethernet is pretty much enough play HD 720P content as if it plays locally. When testing with 54M Wifi (which has much higher latency), a DVD quality video can still play smoothly, but sometimes there will be some noticeable frame drops.

Test Scenario

Here is a test scenario that is known to work:

Download Big Buck Bunny in 854×480 resolution and H.264 format. By default, Windows Server 2008 R2 SP1 won’t be able to play the video. Install K-Lite Codec Pack Basic to get the proper codecs.

Once you know that the server is able to play the video, try connecting and playing the video remotely. If it works, you should get something smooth enough like this:

Hacking

Adding a New Codec

A stream has three main attributes in order to be understood by the client: Major Type, Sub Type and Format Type.

Major Type

There are currently only two major types: video and audio. They are defined as fixed GUID, and mapped to constants defined by our own, in tsmf_constants.h. There could be the third useful major type – subtitle, however it seems that Windows Media Player is not redirecting it even though the subtitle stream is available in the video file.

Sub Type

Sub types are actually the codec needed to decode the stream. Again they are defined as fixed GUID, and we map them to constants in tsmf_constants.h. When the sub types are passed to FFmpeg, the codec constants are mapped again into the FFmpeg codec enum. It’s necessary to separate the mapping into two steps and have our own abstraction layer, since FFmpeg is designed as a sub-plugin and we could support other decoding library in the future.

Format Type

Codecs require information to be able to start decoding properly. For example, the dimension of the video size. Many codecs also require certain codec-specific extra data. In Matroska specification, such extra data is called “CodecPrivate”. In RDP protocol, all those information is packed into a struct called Format Type, defined mostly in MF (Media Foundation) or DirectShow API. Every Format Type is given a fixed GUID and again we map them to our own constants.

Sometimes it could become tricky to retrieve the codec extra data from the Format Type struct. In this case, the Matroska container format might become handy, since it’s much easier to be analyzed by some EBML tree viewer application.

Pixel Format

After a video frame in a stream is decoded, the outcome is a raw image consists of width x height pixels. Those pixels are packed in certain pixel format, usually YUV. Then the image is passed to XVideo directly for hardware decoding and scaling.

The video codecs we currently supported are all producing I420, which is probably the most common format. We also support converting I420 to YV12 in case the video driver does not support it. However, it is yet to investigate whether other pixel formats exist.

Known Issues

The “Landscapes.wtv” file shipped with Windows 7 by default won’t be able to play. It’s difficult to analyze this file format, since FFmpeg does not even able to convert it to mkv. Fast-forwarding is not yet implemented. Volume control is not yet implemented. FAQ

Q: What does “TSMF” mean? A: “TSMF” is the channel name of the Multimedia Redirection protocol, but unfortunately there’s no official explanation in the MS documentation. A reasonable guess would be “Terminal Service Media Foundation”.

Q: I have enough bandwidth and I have registered the tsmf plugin. However my video is very slow like slideshow. A: The video codec is not yet supported, so the video frames are decoded on the server and sent to the client as normal RDP bitmaps. Please look at the standard output for some useful information.

Q: I have registered the tsmf plugin but I cannot hear audio. However if I also registered rdpsnd I can hear it. A: The audio codec is not yet supported, so the audio samples are decoded on the server and sent to the client through rdpsnd plugin. If rdpsnd is not registered, you will not be able to hear anything. But note that even you can hear sound through rdpsnd, video/audio will be out of sync. So the best solution is to request the audio codec support.

You can use FreeRDP with Multimedia Redirection

FreeRDP: A Remote Desktop Protocol Implementation

FreeRDP is a free implementation of the Remote Desktop Protocol (RDP), released under the Apache license. Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience.

Resources

Project website: http://www.freerdp.com/
Issue tracker: https://github.com/FreeRDP/FreeRDP/issues
Sources: https://github.com/FreeRDP/FreeRDP/
Wiki: https://github.com/FreeRDP/FreeRDP/wiki
Downloads and other resources: http://pub.freerdp.com
API doc: http://pub.freerdp.com/api/

IRC channel: #freerdp @ irc.freenode.net Mailing list: https://lists.sourceforge.net/lists/listinfo/freerdp-devel

Microsoft Open Specifications

Information regarding the Microsoft Open Specifications can be found at: http://www.microsoft.com/openspecifications/

A list of reference documentation is maintained here: https://github.com/FreeRDP/FreeRDP/wiki/Reference-Documentation

Compilation

Instructions on how to get started compiling FreeRDP can be found on the wiki: https://github.com/FreeRDP/FreeRDP/wiki/Compilation

https://github.com/FreeRDP/FreeRDP/blob/master/README

Multimedia Redirection

https://github.com/FreeRDP/FreeRDP/wiki/Multimedia-Redirection

2 Found a better solution to the problem
source | link

FreeRDP: A Remote Desktop Protocol Implementation

JACK may be what you've been looking for FreeRDP is a free implementation of the Remote Desktop Protocol (RDP), released under the Apache license. Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience.

Resources

Jack Project website: http://www.freerdp.com/
Issue tracker: https://github.com/FreeRDP/FreeRDP/issues
Sources: https://github.com/FreeRDP/FreeRDP/
Wiki: https://github.com/FreeRDP/FreeRDP/wiki
Downloads and other resources: http://pub.freerdp.com
API doc: http://pub.freerdp.com/api/

JACKIRC channel: #freerdp @ irc.freenode.net
Mailing list: https://lists.sourceforge.net/lists/listinfo/freerdp-devel

Microsoft Open Specifications

Information regarding the Microsoft Open Specifications can be found at: http://www.microsoft.com/openspecifications/

A list of reference documentation is system for handling realmaintained here: https://github.com/FreeRDP/FreeRDP/wiki/Reference-Documentation

Compilation

Instructions on how to get started compiling FreeRDP can be found on the wiki: https://github.com/FreeRDP/FreeRDP/wiki/Compilation

Multimedia Redirection According to [MS-timeRDPEV], low latencyMultimedia Redirection is “used to transfer synchronized audio and video data from a terminal server to a terminal client. The client can play the audio and video data and synchronize this data by using the timing information provided by this protocol”. It’s also called “Video Redirection”.

Usage

Server Requirement

The Multimedia Redirection requires the server running the following Windows version:

Windows 2008 R2: audio and video playback redirection must be enabled manually. Windows 7 Ultimate/Enterprise: works out of the box. Please refer to this blog regarding how to configure Windows 2008 R2.

Note that currently only Windows Media Player supports this feature.

Compilation

The following development packages must be installed to have full Multimedia Redirection features compiled in.

FFmpeg (and MIDIlibavcodec-dev). It runs on GNU ALSA (libasound2-dev) and/Linuxor PulseAudio (libpulse-dev) XVideo (libxv-dev) Quick Start

To use the feature, Solarisregister the “tsmf” dynamic virtual channel as following:

xfreerdp --plugin drdynvc --data tsmf -- (server)

Audio Device

By default, FreeBSDFreeRDP will detect PulseAudio availability and then falls back to ALSA, OS X and Windowsthen use the default audio device. You can force an audio backend and optionally a sound device by adding additional arguments after tsmf. The following example forces to use the default audio device with PulseAudio:

xfreerdp --plugin drdynvc --data tsmf:audio:pulse -- (and can be portedserver)

The following example forces to other POSIXuse the audio device “plughw:0,0” with ALSA:

xfreerdp -conformant platforms-plugin drdynvc --data tsmf:audio:alsa:plughw:0,0 -- (server)

Note that you do not need to register the rdpsnd plugin to have audio playback. ItThe tsmf plugin generally gives higher audio quality than rdpsnd.

XVideo Adaptor

By default, FreeRDP will choose the base port of the last available XVideo adaptor, which should just work in most cases. However sometimes you might want to manually choose a different port, for example, the default adaptor does not work well, or the port is being used by other process. You can connectspecify the —xv-port argument:

xfreerdp --xv-port (port) --plugin drdynvc --data tsmf -- (server)

For a numbercomplete list of different applications to anall available adaptors and ports in your system, run “xvinfo”.

Demuxing and Decoding

Every video file is basically constructed in two steps. Firstly, raw video and audio devicestreams are encoded using some kind of algorithms, called “codec”, such as wellH264, WVC1, MP3, AC3, etc. This step is called “Encoding”. Then the encoded streams are packed together into a single file using specific container format, such as allowing themMatroska(mkv), AVI, MP4, etc. This step is called “Muxing”.

So in order to shareplay a video file, two steps called “Demuxing” and “Decoding” are performed respetively. Those are done transparently when the video is played locally. However when we use Multimedia Redirection, it’s very important to understand that those two steps are performed on different host and different stage: demuxing is performed by the server first, then decoding is performed by the client afterwards. When a video file is opened on the server, the media player will first parse the video file container, retrieve all necessary header information, and separate the content of all video and audio between themselvesstreams. Its clients can runThen the server sends them to the client. The client needs to prepare necessary codec to decode the streams and play them.

It’s obvious that in their own processes (ieorder to be able to redirect specific video file, the server MUST understand the container format. as normal applications)For example, or can they can run withinby default Window Media Player does not support Matroska. You must install the JACKMatroska filter on the server to play mkv files. However, the server does not need to have the codec installed, since once demuxing is done, the original stream content are sent to the client for decoding.

Supported Codecs

Currently the following codecs are supported and tested.

Video:

WVC1 WMV3 H264/AVC1 Audio:

WMA2 MP3 AAC AC3 Bandwidth

Based on some observations, 100M ethernet is pretty much enough play HD 720P content as if it plays locally. When testing with 54M Wifi (iewhich has much higher latency), a DVD quality video can still play smoothly, but sometimes there will be some noticeable frame drops. as

Test Scenario

Here is a "plugin")test scenario that is known to work:

Download Big Buck Bunny in 854×480 resolution and H. JACK also264 format. By default, Windows Server 2008 R2 SP1 won’t be able to play the video. Install K-Lite Codec Pack Basic to get the proper codecs.

Once you know that the server is able to play the video, try connecting and playing the video remotely. If it works, you should get something smooth enough like this:

Hacking

Adding a New Codec

A stream has support for distributingthree main attributes in order to be understood by the client: Major Type, Sub Type and Format Type.

Major Type

There are currently only two major types: video and audio processing across a network. They are defined as fixed GUID, both fast & reliable LANsand mapped to constants defined by our own, in tsmf_constants.h. There could be the third useful major type – subtitle, however it seems that Windows Media Player is not redirecting it even though the subtitle stream is available in the video file.

Sub Type

Sub types are actually the codec needed to decode the stream. Again they are defined as wellfixed GUID, and we map them to constants in tsmf_constants.h. When the sub types are passed to FFmpeg, the codec constants are mapped again into the FFmpeg codec enum. It’s necessary to separate the mapping into two steps and have our own abstraction layer, since FFmpeg is designed as slowera sub-plugin and we could support other decoding library in the future.

Format Type

Codecs require information to be able to start decoding properly. For example, less reliable WANsthe dimension of the video size. Many codecs also require certain codec-specific extra data. In Matroska specification, such extra data is called “CodecPrivate”. In RDP protocol, all those information is packed into a struct called Format Type, defined mostly in MF (Media Foundation) or DirectShow API. Every Format Type is given a fixed GUID and again we map them to our own constants.

JACK was designedSometimes it could become tricky to retrieve the codec extra data from the ground upFormat Type struct. In this case, the Matroska container format might become handy, since it’s much easier to be analyzed by some EBML tree viewer application.

Pixel Format

After a video frame in a stream is decoded, the outcome is a raw image consists of width x height pixels. Those pixels are packed in certain pixel format, usually YUV. Then the image is passed to XVideo directly for professional audio workhardware decoding and scaling.

The video codecs we currently supported are all producing I420, which is probably the most common format. We also support converting I420 to YV12 in case the video driver does not support it. However, it is yet to investigate whether other pixel formats exist.

Known Issues

The “Landscapes.wtv” file shipped with Windows 7 by default won’t be able to play. It’s difficult to analyze this file format, since FFmpeg does not even able to convert it to mkv. Fast-forwarding is not yet implemented. Volume control is not yet implemented. FAQ

Q: What does “TSMF” mean? A: “TSMF” is the channel name of the Multimedia Redirection protocol, but unfortunately there’s no official explanation in the MS documentation. A reasonable guess would be “Terminal Service Media Foundation”.

Q: I have enough bandwidth and its design focusesI have registered the tsmf plugin. However my video is very slow like slideshow. A: The video codec is not yet supported, so the video frames are decoded on two key areasthe server and sent to the client as normal RDP bitmaps. Please look at the standard output for some useful information.

Q: synchronous execution of all clientsI have registered the tsmf plugin but I cannot hear audio. However if I also registered rdpsnd I can hear it. A: The audio codec is not yet supported, so the audio samples are decoded on the server and low latency operationsent to the client through rdpsnd plugin. If rdpsnd is not registered, you will not be able to hear anything. But note that even you can hear sound through rdpsnd, video/audio will be out of sync. So the best solution is to request the audio codec support.

JACK may be what you've been looking for

Jack

JACK is system for handling real-time, low latency audio (and MIDI). It runs on GNU/Linux, Solaris, FreeBSD, OS X and Windows (and can be ported to other POSIX-conformant platforms). It can connect a number of different applications to an audio device, as well as allowing them to share audio between themselves. Its clients can run in their own processes (ie. as normal applications), or can they can run within the JACK server (ie. as a "plugin"). JACK also has support for distributing audio processing across a network, both fast & reliable LANs as well as slower, less reliable WANs.

JACK was designed from the ground up for professional audio work, and its design focuses on two key areas: synchronous execution of all clients, and low latency operation

FreeRDP: A Remote Desktop Protocol Implementation

FreeRDP is a free implementation of the Remote Desktop Protocol (RDP), released under the Apache license. Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience.

Resources

Project website: http://www.freerdp.com/
Issue tracker: https://github.com/FreeRDP/FreeRDP/issues
Sources: https://github.com/FreeRDP/FreeRDP/
Wiki: https://github.com/FreeRDP/FreeRDP/wiki
Downloads and other resources: http://pub.freerdp.com
API doc: http://pub.freerdp.com/api/

IRC channel: #freerdp @ irc.freenode.net
Mailing list: https://lists.sourceforge.net/lists/listinfo/freerdp-devel

Microsoft Open Specifications

Information regarding the Microsoft Open Specifications can be found at: http://www.microsoft.com/openspecifications/

A list of reference documentation is maintained here: https://github.com/FreeRDP/FreeRDP/wiki/Reference-Documentation

Compilation

Instructions on how to get started compiling FreeRDP can be found on the wiki: https://github.com/FreeRDP/FreeRDP/wiki/Compilation

Multimedia Redirection According to [MS-RDPEV], Multimedia Redirection is “used to transfer synchronized audio and video data from a terminal server to a terminal client. The client can play the audio and video data and synchronize this data by using the timing information provided by this protocol”. It’s also called “Video Redirection”.

Usage

Server Requirement

The Multimedia Redirection requires the server running the following Windows version:

Windows 2008 R2: audio and video playback redirection must be enabled manually. Windows 7 Ultimate/Enterprise: works out of the box. Please refer to this blog regarding how to configure Windows 2008 R2.

Note that currently only Windows Media Player supports this feature.

Compilation

The following development packages must be installed to have full Multimedia Redirection features compiled in.

FFmpeg (libavcodec-dev) ALSA (libasound2-dev) and/or PulseAudio (libpulse-dev) XVideo (libxv-dev) Quick Start

To use the feature, register the “tsmf” dynamic virtual channel as following:

xfreerdp --plugin drdynvc --data tsmf -- (server)

Audio Device

By default, FreeRDP will detect PulseAudio availability and then falls back to ALSA, and then use the default audio device. You can force an audio backend and optionally a sound device by adding additional arguments after tsmf. The following example forces to use the default audio device with PulseAudio:

xfreerdp --plugin drdynvc --data tsmf:audio:pulse -- (server)

The following example forces to use the audio device “plughw:0,0” with ALSA:

xfreerdp --plugin drdynvc --data tsmf:audio:alsa:plughw:0,0 -- (server)

Note that you do not need to register the rdpsnd plugin to have audio playback. The tsmf plugin generally gives higher audio quality than rdpsnd.

XVideo Adaptor

By default, FreeRDP will choose the base port of the last available XVideo adaptor, which should just work in most cases. However sometimes you might want to manually choose a different port, for example, the default adaptor does not work well, or the port is being used by other process. You can specify the —xv-port argument:

xfreerdp --xv-port (port) --plugin drdynvc --data tsmf -- (server)

For a complete list of all available adaptors and ports in your system, run “xvinfo”.

Demuxing and Decoding

Every video file is basically constructed in two steps. Firstly, raw video and audio streams are encoded using some kind of algorithms, called “codec”, such as H264, WVC1, MP3, AC3, etc. This step is called “Encoding”. Then the encoded streams are packed together into a single file using specific container format, such as Matroska(mkv), AVI, MP4, etc. This step is called “Muxing”.

So in order to play a video file, two steps called “Demuxing” and “Decoding” are performed respetively. Those are done transparently when the video is played locally. However when we use Multimedia Redirection, it’s very important to understand that those two steps are performed on different host and different stage: demuxing is performed by the server first, then decoding is performed by the client afterwards. When a video file is opened on the server, the media player will first parse the video file container, retrieve all necessary header information, and separate the content of all video and audio streams. Then the server sends them to the client. The client needs to prepare necessary codec to decode the streams and play them.

It’s obvious that in order to be able to redirect specific video file, the server MUST understand the container format. For example, by default Window Media Player does not support Matroska. You must install the Matroska filter on the server to play mkv files. However, the server does not need to have the codec installed, since once demuxing is done, the original stream content are sent to the client for decoding.

Supported Codecs

Currently the following codecs are supported and tested.

Video:

WVC1 WMV3 H264/AVC1 Audio:

WMA2 MP3 AAC AC3 Bandwidth

Based on some observations, 100M ethernet is pretty much enough play HD 720P content as if it plays locally. When testing with 54M Wifi (which has much higher latency), a DVD quality video can still play smoothly, but sometimes there will be some noticeable frame drops.

Test Scenario

Here is a test scenario that is known to work:

Download Big Buck Bunny in 854×480 resolution and H.264 format. By default, Windows Server 2008 R2 SP1 won’t be able to play the video. Install K-Lite Codec Pack Basic to get the proper codecs.

Once you know that the server is able to play the video, try connecting and playing the video remotely. If it works, you should get something smooth enough like this:

Hacking

Adding a New Codec

A stream has three main attributes in order to be understood by the client: Major Type, Sub Type and Format Type.

Major Type

There are currently only two major types: video and audio. They are defined as fixed GUID, and mapped to constants defined by our own, in tsmf_constants.h. There could be the third useful major type – subtitle, however it seems that Windows Media Player is not redirecting it even though the subtitle stream is available in the video file.

Sub Type

Sub types are actually the codec needed to decode the stream. Again they are defined as fixed GUID, and we map them to constants in tsmf_constants.h. When the sub types are passed to FFmpeg, the codec constants are mapped again into the FFmpeg codec enum. It’s necessary to separate the mapping into two steps and have our own abstraction layer, since FFmpeg is designed as a sub-plugin and we could support other decoding library in the future.

Format Type

Codecs require information to be able to start decoding properly. For example, the dimension of the video size. Many codecs also require certain codec-specific extra data. In Matroska specification, such extra data is called “CodecPrivate”. In RDP protocol, all those information is packed into a struct called Format Type, defined mostly in MF (Media Foundation) or DirectShow API. Every Format Type is given a fixed GUID and again we map them to our own constants.

Sometimes it could become tricky to retrieve the codec extra data from the Format Type struct. In this case, the Matroska container format might become handy, since it’s much easier to be analyzed by some EBML tree viewer application.

Pixel Format

After a video frame in a stream is decoded, the outcome is a raw image consists of width x height pixels. Those pixels are packed in certain pixel format, usually YUV. Then the image is passed to XVideo directly for hardware decoding and scaling.

The video codecs we currently supported are all producing I420, which is probably the most common format. We also support converting I420 to YV12 in case the video driver does not support it. However, it is yet to investigate whether other pixel formats exist.

Known Issues

The “Landscapes.wtv” file shipped with Windows 7 by default won’t be able to play. It’s difficult to analyze this file format, since FFmpeg does not even able to convert it to mkv. Fast-forwarding is not yet implemented. Volume control is not yet implemented. FAQ

Q: What does “TSMF” mean? A: “TSMF” is the channel name of the Multimedia Redirection protocol, but unfortunately there’s no official explanation in the MS documentation. A reasonable guess would be “Terminal Service Media Foundation”.

Q: I have enough bandwidth and I have registered the tsmf plugin. However my video is very slow like slideshow. A: The video codec is not yet supported, so the video frames are decoded on the server and sent to the client as normal RDP bitmaps. Please look at the standard output for some useful information.

Q: I have registered the tsmf plugin but I cannot hear audio. However if I also registered rdpsnd I can hear it. A: The audio codec is not yet supported, so the audio samples are decoded on the server and sent to the client through rdpsnd plugin. If rdpsnd is not registered, you will not be able to hear anything. But note that even you can hear sound through rdpsnd, video/audio will be out of sync. So the best solution is to request the audio codec support.

1
source | link

JACK may be what you've been looking for

Jack

JACK is system for handling real-time, low latency audio (and MIDI). It runs on GNU/Linux, Solaris, FreeBSD, OS X and Windows (and can be ported to other POSIX-conformant platforms). It can connect a number of different applications to an audio device, as well as allowing them to share audio between themselves. Its clients can run in their own processes (ie. as normal applications), or can they can run within the JACK server (ie. as a "plugin"). JACK also has support for distributing audio processing across a network, both fast & reliable LANs as well as slower, less reliable WANs.

JACK was designed from the ground up for professional audio work, and its design focuses on two key areas: synchronous execution of all clients, and low latency operation