mirror of
https://github.com/mumble-voip/mumble.git
synced 2025-10-26 11:19:16 +00:00
d05262a9ce
Integrates our network protocol docs [1][2]. This will allow us to drop the separate repository. Git history is preserved through adding/merging with/through git subtree. via ``` git subtree add --prefix docs/dev/network-protocol https://github.com/mumble-voip/mumble-protocol.git master ``` [1]: https://github.com/mumble-voip/mumble-protocol [2]: https://mumble-protocol.readthedocs.io/ --- Add 'docs/dev/network-protocol/' from commit 'b0d142143c5c37797d787fd7f71eb172dd1219ad' git-subtree-dir: docs/dev/network-protocol git-subtree-mainline:65462b4c3bgit-subtree-split:b0d142143c
394 lines
20 KiB
ReStructuredText
394 lines
20 KiB
ReStructuredText
.. _voice-data:
|
|
|
|
Voice data
|
|
==========
|
|
|
|
Mumble audio channel is used to transmit the actual audio packets over the
|
|
network. Unlike the TCP control channel, the audio channel uses a custom
|
|
encoding for the audio packets. The audio channel is transport independent and
|
|
features such as encryption are implemented by the transport layer. Integers
|
|
above 8-bits are encoded using the `Variable length integer encoding`_.
|
|
|
|
.. _packet-format:
|
|
|
|
Packet format
|
|
-------------
|
|
|
|
The mumble audio channel packets are variable length packets that begin with an
|
|
8-bit header field which describes the packet type and target. The most
|
|
significant 3 bits define the packet type while the remaining 5 bits define the
|
|
target. The header is followed by the packet payload. The maximum size for the
|
|
whole audio data packet is 1020 bytes. This allows applications to use 1024
|
|
byte buffers for receiving UDP datagrams with the 4-byte encryption header
|
|
overhead.
|
|
|
|
.. _Audio packet structure:
|
|
.. table:: Audio packet structure
|
|
:class: bits8
|
|
|
|
+-------------------------------+
|
|
| Audio packet structure |
|
|
+===+===+===+===+===+===+===+===+
|
|
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
|
|
+---+---+---+---+---+---+---+---+
|
|
| ``type`` | ``target`` |
|
|
+-----------+-------------------+
|
|
| Payload... |
|
|
+-------------------------------+
|
|
|
|
type
|
|
The audio packet type. The packets transmitted over the audio channel are
|
|
either ping packets used to diagnose the transport layer connectivity or
|
|
audio packets encoded with different codecs. Different types are listed in
|
|
`Audio packet types`_ table.
|
|
|
|
.. _Audio packet types:
|
|
.. table:: Audio packet types
|
|
|
|
+---------+---------------+--------------------------------------------+
|
|
| Type | Bitfield | Description |
|
|
+=========+===============+============================================+
|
|
| ``0`` | ``000xxxxx`` | CELT Alpha encoded voice data |
|
|
+---------+---------------+--------------------------------------------+
|
|
| ``1`` | ``001xxxxx`` | Ping packet |
|
|
+---------+---------------+--------------------------------------------+
|
|
| ``2`` | ``010xxxxx`` | Speex encoded voice data |
|
|
+---------+---------------+--------------------------------------------+
|
|
| ``3`` | ``011xxxxx`` | CELT Beta encoded voice data |
|
|
+---------+---------------+--------------------------------------------+
|
|
| ``4`` | ``100xxxxx`` | OPUS encoded voice data |
|
|
+---------+---------------+--------------------------------------------+
|
|
| ``5-7`` | | Unused |
|
|
+---------+---------------+--------------------------------------------+
|
|
|
|
target
|
|
The target portion defines the recipient for the audio data. The two constant
|
|
targets are *Normal talking* (``0``) and *Server Loopback* (``31``). The
|
|
range 1-30 is reserved for whisper targets. These targets are specified
|
|
separately in the control channel using the ``VoiceTarget`` packets. The
|
|
targets are listed in `Audio targets`_ table.
|
|
|
|
When a client registers a VoiceTarget on the server, it gives the target an
|
|
ID. This voice target ID can be used as a target in the voice packets to send
|
|
audio to specific users or channels. When receiving whisper-audio the server
|
|
uses target 1 to specify the audio results from a whisper to a channel and
|
|
target 2 to specify that the audio results from a direct whisper to the user.
|
|
|
|
.. _Audio targets:
|
|
.. table:: Audio targets
|
|
|
|
+-----------+-----------------------------------------------------+
|
|
| Target | Description |
|
|
+===========+=====================================================+
|
|
| ``0`` | Normal talking |
|
|
+-----------+-----------------------------------------------------+
|
|
| ``1-30`` | Whisper target |
|
|
| | |
|
|
| | - VoiceTarget ID when sending whisper from client. |
|
|
| | - 1 when receiving whisper to channel. |
|
|
| | - 2 when receiving direct whisper to user. |
|
|
+-----------+-----------------------------------------------------+
|
|
| ``31`` | Server loopback |
|
|
+-----------+-----------------------------------------------------+
|
|
|
|
Ping packet
|
|
~~~~~~~~~~~
|
|
|
|
Audio channel ping packets are used as part of the connectivity checks on the
|
|
audio transport layer. These packets contain only varint encoded timestamp as
|
|
data. See `UDP connectivity checks`_ section below for the logic involved in
|
|
the connectivity checks.
|
|
|
|
.. _Audio transport ping packet:
|
|
|
|
.. table:: Audio transport ping packet
|
|
|
|
+------------+-------------+----------------------------------+
|
|
| Field | Type | Description |
|
|
+============+=============+==================================+
|
|
| Header | ``byte`` | ``00100000b`` (``0x20``) |
|
|
+------------+-------------+----------------------------------+
|
|
| Data | ``varint`` | Timestamp |
|
|
+------------+-------------+----------------------------------+
|
|
|
|
Header
|
|
Common audio packet header. For ping packets this should have the value of
|
|
0x20.
|
|
|
|
Data
|
|
Timestamp. The packet should be echoed back so the timestamp format can be
|
|
decided by the original sender - the only limitation is that it must fit in a
|
|
64-bit integer for the varint encoding.
|
|
|
|
Encoded audio data packet
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Encoded audio packets contain the actual user audio data for the voice
|
|
communication. Incoming audio data packets contain the common header byte
|
|
followed by varint encoded session ID of the source user and varint encoded
|
|
sequence number of the packet. Outgoing audio data packets contain only the
|
|
header byte and the sequence number of the packet. The server matches these to
|
|
the correct session using the transport layer information.
|
|
|
|
The remainder of the packet is made up of multiple encoded audio segments and
|
|
optional positional audio information. The audio segment format depends on the
|
|
codec of the whole audio packets. The audio segments contain codec
|
|
implementation specific information on where the audio segments end so the
|
|
possible positional audio data can be read from the end.
|
|
|
|
.. _Incoming encoded audio packet:
|
|
.. table:: Incoming encoded audio packet
|
|
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Field | Type | Description |
|
|
+====================+==============+===========================================================+
|
|
| Header | ``byte`` | Codec type/Audio target |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Session ID | ``varint`` | Session ID of the source user. |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Sequence Number | ``varint`` | Sequence number of the first audio data **segment**. |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Payload | ``byte[]`` | Audio payload |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Position Info | ``float[3]`` | Positional audio information |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
|
|
|
|
.. _Outgoing encoded audio packet:
|
|
.. table:: Outgoing encoded audio packet
|
|
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Field | Type | Description |
|
|
+====================+==============+===========================================================+
|
|
| Header | ``byte`` | Codec type/Audio target |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Sequence Number | ``varint`` | Sequence number of the first audio data **segment**. |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Payload | ``byte[]`` | Audio payload |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
| Position Info | ``float[3]`` | Positional audio information |
|
|
+--------------------+--------------+-----------------------------------------------------------+
|
|
|
|
Header
|
|
The common audio packet header
|
|
|
|
Session ID
|
|
Session ID of the user to whom the audio packet belongs.
|
|
|
|
Sequence Number
|
|
Audio data sequence number. The sequence number is used to maintain the
|
|
packet order when the audio data is transported over unreliable transports
|
|
such as UDP.
|
|
|
|
The sequence number might increase by more than one between subsequent audio
|
|
packets in case the audio packets contain multiple audio segments. This
|
|
allows the packet loss concealment algorithms to figure out how many audio
|
|
frames were lost between two received packets.
|
|
|
|
Payload
|
|
Audio payload. Format depends on the audio codec defined in the Header. The
|
|
payload must be self-delimiting to determine whether the position info exists
|
|
at the end of the packet.
|
|
|
|
Position Info
|
|
The XYZ coordinates of the audio source. In addition to sending the position
|
|
information, the user must be using a positional plugin defined in the
|
|
``UserState`` message. The plugins might define different contexts which
|
|
prevent voice communication between users in other contexts.
|
|
|
|
Speex and CELT audio frames
|
|
"""""""""""""""""""""""""""
|
|
|
|
Encoded Speex and CELT audio is transported as individual encoded frames. Each
|
|
frame is prefixed with a single byte length and terminator header.
|
|
|
|
.. _celt-encoded-audio-data:
|
|
|
|
.. table:: CELT encoded audio data
|
|
|
|
+---------+-------------+-----------------------------------------+
|
|
| Field | Type | Description |
|
|
+=========+=============+=========================================+
|
|
| Header | ``byte`` | length/continuation header |
|
|
+---------+-------------+-----------------------------------------+
|
|
| Data | ``byte[]`` | Encoded voice frame |
|
|
+---------+-------------+-----------------------------------------+
|
|
|
|
Header
|
|
The length of the Data field. The most significant bit (``0x80``) acts as the
|
|
continuation bit and is set for all but the last frame in the payload. The
|
|
remaining 7 bits of the header contain the actual length of the Data frame.
|
|
|
|
Note the length may be zero, which is used to signal the end of a voice
|
|
transmission. In this case the audio data is a single zero-byte which can be
|
|
interpreted normally as length of 0 with no continuation bit set.
|
|
|
|
Data
|
|
Single encoded audio frame. The encoding depends on the codec ``type`` header
|
|
of the whole audio packet
|
|
|
|
Opus audio frames
|
|
"""""""""""""""""
|
|
|
|
Encoded Opus audio is transported as a single Opus audio frame. The frame is prefixed with a variable byte header.
|
|
|
|
.. _opus-encoded-audio-data:
|
|
|
|
.. table:: Opus encoded audio data
|
|
|
|
+---------+-------------+-----------------------------------------+
|
|
| Field | Type | Description |
|
|
+=========+=============+=========================================+
|
|
| Header | ``varint`` | length/terminator header |
|
|
+---------+-------------+-----------------------------------------+
|
|
| Data | ``byte[]`` | Encoded voice frame |
|
|
+---------+-------------+-----------------------------------------+
|
|
|
|
Header
|
|
The length of the Data field. 16-bit variable length integer encoded length
|
|
and terminator bit value. The varint encoding is the same as with 64-bit
|
|
values, but only 16-bit unencoded values are allowed.
|
|
|
|
The maximum voice frame size is 8191 (``0x1FFF``) bytes requiring the 13 least
|
|
significant bits of the header. The 14th bit (mask: ``0x2000``) is the terminator
|
|
bit which signals whether the packet is the last one in the voice
|
|
transmission.
|
|
|
|
Note: In CELT the "continuation bit" in the header defines whether there are
|
|
more audio frames in the current packet. Opus always contains only one frame
|
|
in the packet. In CELT the voice transmission end is signaled with a
|
|
zero-byte CELT packet while in Opus we have a dedicated termination bit in
|
|
the header.
|
|
|
|
Data
|
|
The encoded Opus data.
|
|
|
|
Codecs
|
|
------
|
|
|
|
Mumble supports three distinct codecs; Older Mumble versions use Speex for low
|
|
bitrate audio and CELT for higher quality audio while new Mumble versions
|
|
prefer Opus for all audio. When multiple clients with different capabilities
|
|
communicate together the server is responsible for resolving the codec to use.
|
|
The clients should respect the server resolution if they are capable.
|
|
|
|
If the server resolves a codec a client doesn't support, that client is free to
|
|
use any codec it prefers. Usually this means the client will not be able to
|
|
decode incoming audio, but it can still send encoded audio out.
|
|
|
|
The CELT bitstream was never frozen which makes most CELT versions incompatible
|
|
with each other. The two CELT bitstreams supported by Mumble are: CELT 0.7.0
|
|
(CELT Alpha) and CELT 0.11.0 (CELT Beta). While CELT 0.7.0 should technically
|
|
be supported by most Mumble implementations, some servers might be configured
|
|
to force Opus codec for the users. Mumble has had Opus support since 1.2.4
|
|
(June 2013) so it should be safe to assume most clients in use support this
|
|
now.
|
|
|
|
Whispering
|
|
----------
|
|
|
|
Normal talking can be heard by the users of the current channel and all linked
|
|
channels as long as the speaker has Talk permission on these channels. If the
|
|
speaker wishes to broadcast the voice to specific users or channels, they may
|
|
use whispering. This is achieved by registering a voice target using the
|
|
VoiceTarget message and specifying the target ID as the target in the first
|
|
byte of the UDP packet.
|
|
|
|
UDP connectivity checks
|
|
-----------------------
|
|
|
|
Since UDP is a connectionless protocol, it is heavily affected by network
|
|
topology such as NAT configuration. It should not be used for audio
|
|
transmission before the connectivity has been determined.
|
|
|
|
The client starts the connectivity checks by sending a `Ping packet`_ to the
|
|
server. When the server receives this packet it will respond by echoing it back
|
|
to the address it received it from. Once the client receives the response from
|
|
the server it can start using the UDP transport for audio data. When the server
|
|
receives incoming audio data over the UDP transport it can switch the outgoing
|
|
audio over to UDP transport as well.
|
|
|
|
If the client stops receiving replies to the UDP pings at some point, it should
|
|
start tunneling the voice communication through the TCP tunnel as described in
|
|
the `Tunneling audio over TCP`_ below. When the server receives a tunneled
|
|
packet over the TCP connection it must also stop using the UDP for
|
|
communication. The client should still continue sending audio ping packets over
|
|
the UDP transport in case the UDP connection is restored and the communication
|
|
can be switched back to it.
|
|
|
|
Tunneling audio over TCP
|
|
------------------------
|
|
|
|
If the UDP channel isn't available the voice packets can be transmitted through
|
|
the TCP transport used for the control channel. These messages use the normal
|
|
TCP prefixing, as shown in figure :ref:`mumble-packet`: 16-bit message type
|
|
followed by 32-bit message length. However unlike other TCP messages, the audio
|
|
packets are not encoded as protocol buffer messages but instead the raw audio
|
|
packet described in `Packet format`_ should be written to the TCP socket
|
|
verbatim.
|
|
|
|
When the packets are received it is safe to parse the type and length fields
|
|
normally. If the type matches that of the audio tunnel the rest of the message
|
|
should be processed as an UDP packet without attempting a protocol buffer
|
|
decoding.
|
|
|
|
Implementation note
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
When implementing the protocol it is easier to ignore the UDP transfer layer at
|
|
first and just tunnel the UDP data through the TCP tunnel. The TCP layer must
|
|
be implemented for authentication in any case. Making sure that the voice
|
|
transmission works before implementing the UDP protocol simplifies debugging
|
|
greatly.
|
|
|
|
Encryption
|
|
----------
|
|
|
|
All the packets are encrypted once during transfer. The actual encryption
|
|
depends on the used transport layer. If the packets are tunneled through TCP
|
|
they are encrypted using the TLS that encrypts the whole control channel
|
|
connection and if they are sent directly using UDP they must be encrypted using
|
|
the OCB-AES128 encryption.
|
|
|
|
Variable length integer encoding
|
|
--------------------------------
|
|
|
|
The variable length integer encoding (``varint``) is used to encode long,
|
|
64-bit, integers so that short values do not need the full 8 bytes to be
|
|
transferred. The basic idea behind the encoding is prefixing the value with a
|
|
length prefix and then removing the leading zeroes from the value. The positive
|
|
numbers are always right justified. That is to say that the least significant
|
|
bit in the encoded presentation matches the least significant bit in the
|
|
decoded presentation. The *varint prefixes* table contains the definitions of
|
|
the different length prefixes. The encoded ``x`` bits are part of the decoded
|
|
number while the ``_`` signifies a unused bit. Encoding should be done by
|
|
searching the first decoded description that fits the number that should be
|
|
decoded, truncating it to the required bytes and combining it with the defined
|
|
encoding prefix.
|
|
|
|
See the *quint64* shift operators in
|
|
https://github.com/mumble-voip/mumble/blob/master/src/PacketDataStream.h
|
|
for a reference implementation.
|
|
|
|
.. table:: Varint prefixes
|
|
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| Encoded | Decoded |
|
|
+==================================+========================================================+
|
|
| ``0xxxxxxx`` | 7-bit positive number |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``10xxxxxx`` + 1 byte | 14-bit positive number |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``110xxxxx`` + 2 bytes | 21-bit positive number |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``1110xxxx`` + 3 bytes | 28-bit positive number |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``111100__`` + ``int`` (32-bit) | 32-bit positive number |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``111101__`` + ``long`` (64-bit) | 64-bit number |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``111110__`` + ``varint`` | Negative recursive varint |
|
|
+----------------------------------+--------------------------------------------------------+
|
|
| ``111111xx`` | Byte-inverted negative two bit number (``~xx``) |
|
|
+----------------------------------+--------------------------------------------------------+
|