US6501797B1 - System and method for improved fine granular scalable video using base layer coding information - Google Patents

System and method for improved fine granular scalable video using base layer coding information Download PDF

Info

Publication number
US6501797B1
US6501797B1 US09/347,881 US34788199A US6501797B1 US 6501797 B1 US6501797 B1 US 6501797B1 US 34788199 A US34788199 A US 34788199A US 6501797 B1 US6501797 B1 US 6501797B1
Authority
US
United States
Prior art keywords
base layer
video
video frames
parameter
bit rate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
US09/347,881
Inventor
Mihaela van der Schaar
Yingwei Chen
Hayder Radha
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Funai Electric Co Ltd
Original Assignee
Koninklijke Philips Electronics NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics NV filed Critical Koninklijke Philips Electronics NV
Priority to US09/347,881 priority Critical patent/US6501797B1/en
Assigned to PHILIPS ELECTRONICS NORTH AMERICA CORPORATION reassignment PHILIPS ELECTRONICS NORTH AMERICA CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHEN, YINGWEI, RADHA, HAYDER, VAN DER SCHAAR, MIHAELA
Priority to JP2001508171A priority patent/JP2003533067A/en
Priority to PCT/EP2000/006243 priority patent/WO2001003441A1/en
Priority to EP00952999A priority patent/EP1110405A1/en
Priority to CNB00801843XA priority patent/CN1192629C/en
Priority to KR1020017002917A priority patent/KR20010086365A/en
Priority to AU65609/00A priority patent/AU6560900A/en
Assigned to KONINKLIJIKE PHILIPS ELECTRONIS N.V. reassignment KONINKLIJIKE PHILIPS ELECTRONIS N.V. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PHILIPS ELECTRONICS NORTH AMERICA CORPORTION
Publication of US6501797B1 publication Critical patent/US6501797B1/en
Application granted granted Critical
Assigned to IPG ELECTRONICS 503 LIMITED reassignment IPG ELECTRONICS 503 LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KONINKLIJKE PHILIPS ELECTRONICS N.V.
Assigned to FUNAI ELECTRIC CO., LTD. reassignment FUNAI ELECTRIC CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: IPG ELECTRONICS 503 LIMITED
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/12Systems in which the television signal is transmitted via one channel or a plurality of parallel channels, the bandwidth of each channel being less than the bandwidth of the television signal
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/34Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103Selection of coding mode or of prediction mode
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136Incoming video signal characteristics or properties
    • H04N19/137Motion inside a coding unit, e.g. average field, frame or block difference
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146Data rate or code amount at the encoder output
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/187Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H04N19/29Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding involving scalability at the object level, e.g. video object layer [VOL]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/33Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the spatial domain
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/37Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability with arrangements for assigning different transmission priorities to video input data or to video coded data

Definitions

  • the present invention is related to that disclosed in United States patent application Ser. No. 09/347,882), entitled “SYSTEM AND METHOD FOR FINE GRANULAR SCALABLE VIDEO WITH SELECTIVE QUALITY ENHANCEMENT,” which is being filed concurrently herewith and is commonly assigned to the assignee of the present invention.
  • the disclosure of the related patent application is incorporated herein by reference for all purposes as if fully set forth herein.
  • the present invention is directed, in general, to video encoding systems and, more specifically, to an encoding system for streaming video data.
  • Real-time streaming of multimedia content over data networks has become an increasingly common application in recent years.
  • a wide range of interactive and non-interactive multimedia applications such as news-on-demand, live network television viewing, video conferencing, among others, rely on end-to-end streaming video techniques.
  • streaming video applications Unlike a “downloaded” video file, which may be retrieved first in “non-real” time and viewed or played back later in “real” time, streaming video applications require a video transmitter that encodes and transmits a video signal over a data network to a video receiver, which must decode and display the video signal in real time.
  • Scalable video coding is a desirable feature for many multimedia applications and services that are used in systems employing decoders with a wide range of processing power. Scalability allows processors with low computational power to decode only a subset of the scalable video stream.
  • Another use of scalable video is in environments with a variable transmission bandwidth. In those environments, receivers with low-access bandwidth receive, and consequently decode, only a subset of the scalable video stream, where the amount of that subset is proportional to the available bandwidth.
  • BL base layer
  • EL enhancement layer
  • the base layer part of the scalable video stream represents, in general, the minimum amount of data needed for decoding that stream.
  • the enhanced layer part of the stream represents additional information, and therefore enhances the video signal representation when decoded by the receiver.
  • the base layer transmission rate may be established at the minimum guaranteed transmission rate of the variable bandwidth system.
  • the base layer rate may be established at 256 kbps also. If the actual available bandwidth is 384 kbps, the extra 128 kbps of bandwidth may be used by the enhancement layer to improve on the basic signal transmitted at the base layer rate.
  • a certain scalability structure For each type of video scalability, a certain scalability structure is identified.
  • the scalability structure defines the relationship among the pictures of the base layer and the pictures of the enhanced layer.
  • One class of scalability is fine-granular scalability. Images coded with this type of scalability can be decoded progressively. In other words, the decoder may decode and display the image with only a subset of the data used for coding that image. As more data is received, the quality of the decoded image is progressively enhanced until the complete information is received, decoded, and displayed.
  • the newly proposed MPEG-4 standard is directed to new video streaming applications based on very low bit rate coding, such as video-phone, mobile multimedia and audio-visual communications, multimedia e-mail, remote sensing, interactive games, and the like.
  • bit rate coding such as video-phone, mobile multimedia and audio-visual communications, multimedia e-mail, remote sensing, interactive games, and the like.
  • FGS fine-granular scalability
  • FGS primarily targets applications where video is streamed over heterogeneous networks in real-time. It provides bandwidth adaptivity by encoding content once for a range of bit rates, and enabling the video transmission server to change the transmission rate dynamically without in-depth knowledge or parsing of the video bit stream.
  • a limitation of the compression scheme currently adopted as reference for FGS resides in its inability to exploit the base layer coding information for improving the compression efficiency of the enhancement-layer.
  • Another disadvantage of currently adopted FGS schemes resides in the fact that enhancement layer frames are coded independently of each other (i.e., “intra” coding of frames).
  • the intra-frame coding of the enhancement layer is necessary for error resilience and for easy bit rate change at transmission time.
  • each enhancement frame is optimally coded in its own context, discontinuity or inconsistency between the image quality of consecutive frames is often introduced.
  • the resulting FGS enhanced video may have “flashing” artifacts across frames. This is particular annoying and highly visible when compared to the more “visually stable” single layer coded video.
  • the proposed encoding technique uses one or more parameters taken from the base layer compression information (e.g., motion vectors, base layer quantization errors, rate-control info, etc.) to improve the image quality of the enhancement layer.
  • the present invention may use single layer coding at multiple bit rates as “guidelines” for FGS encoding.
  • the new compression techniques may be applied independent of the transforms chosen in the base and enhancement layers (e.g., discrete cosine transform (DCT) or wavelets).
  • DCT discrete cosine transform
  • the use of certain base layer or single-layer information is less straightforward if different coding schemes are employed at the base and enhancement layers.
  • a video encoder comprising a base layer circuit capable of receiving an input stream of video frames and generating therefrom compressed base layer video frames suitable for transmission at a base layer bit rate to a streaming video receiver and an enhancement layer circuit capable of receiving the input stream of video frames and a decoded version of the compressed base layer video frames and generating therefrom enhancement layer video data associated with, and allocated to, corresponding ones of the compressed base layer video frames and suitable for transmission at a modifiable enhancement layer bit rate to the streaming video receiver an apparatus for controlling transmission of the enhancement layer video data.
  • the apparatus comprises a base layer parameter monitor capable of receiving at least one base layer parameter and, in response thereto, modifying an allocation of the enhancement layer video data among the corresponding ones of the compressed base layer video frames.
  • the video encoder comprises a motion estimation circuit capable of receiving the input stream of video frames and determining therefrom a base layer motion parameter associated with at least one selected frame sequence in the input stream of video frames.
  • the base layer parameter monitor receives the base layer motion parameter and, in response thereto, modifies the allocation of the enhancement layer video data according to a level of motion in the at least one selected frame sequence indicated by the base layer motion parameter.
  • the video encoder comprises a quantization circuit capable of receiving and quantizing transform data associated with the input stream of video frames to thereby reduce a size of the transform data and further capable of determining a base layer quantization error parameter associated with the quantized transform data.
  • the base layer parameter monitor receives the base layer quantization error parameter and, in response thereto, modifies the allocation of the enhancement layer video data according to a quantization error indicated by the base layer quantization error parameter.
  • the video encoder comprises a base layer rate allocation circuit capable of determining the base layer bit rate, wherein the base layer bit rate is set at a pre-determined minimum rate at which the compressed base layer video frames are transmitted to the streaming video receiver, and generating therefrom a base layer bit rate parameter associated with the base layer bit rate.
  • the base layer parameter monitor receives the base layer bit rate parameter and, in response thereto, modifies the allocation of the enhancement layer video data according to an estimated difference between the compressed base layer video frames and estimated compressed base layer video frames associated with a second base layer bit rate greater than the pre-determined minimum rate.
  • FIG. 1 illustrates an end-to-end transmission of streaming video from a streaming video transmitter through a data network to a streaming video receiver, according to one embodiment of the present invention
  • FIG. 2 illustrates a video encoder in accordance with one embodiment of the prior art
  • FIG. 3 illustrates an exemplary video encoder in accordance with one embodiment of the present invention.
  • FIG. 4 is a flow diagram illustrating the operation of an exemplary video encoder in accordance with one embodiment of the present invention.
  • FIGS. 1 through 4 discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the present invention may be implemented in any suitably arranged video encoder.
  • FIG. 1 illustrates an end-to-end transmission of streaming video from streaming video transmitter 110 through data network 120 to streaming video receiver 130 , according to one embodiment of the present invention.
  • streaming video transmitter 110 may be any one of a wide variety of sources of video frames, including a data network server, a television station, a cable network, a desktop personal computer (PC), or the like.
  • Streaming video transmitter 110 comprises video frame source 112 , video encoder 114 and recoder buffer 116 .
  • Video frame source 112 may be any device capable of generating a sequence of uncompressed video frames, including a television antenna and receiver unit, a video cassette player, a video camera, a disk storage device capable of storing a “raw” video clip, and the like.
  • the uncompressed video frames enter video encoder 114 at a given picture rate (or “streaming rate”) and are compressed according to any known compression algorithm or device, such as an MPEG-4 encoder.
  • Video encoder 114 then transmits the compressed video frames to encoder buffer 116 for buffering in preparation for transmission across data network 120 .
  • Data network 120 may be any suitable network and may include portions of both public data networks, such as the Internet, and private data networks, such as an enterprise-owned local area network (LAN) or wide area network (WAN).
  • LAN local area network
  • WAN wide area network
  • Streaming video receiver 130 comprises decoder buffer 132 , video decoder 134 and video display 136 .
  • Decoder buffer 132 receives and stores streaming compressed video frames from data network 120 . Decoder buffer 132 then transmits the compressed video frames to video decoder 134 as required.
  • Video decoder 134 decompresses the video frames at the same rate (ideally) at which the video frames were compressed by video encoder 114 .
  • Video decoder 134 sends the decompressed frames to video display 136 for play-back on the screen of video display 134 .
  • FIG. 2 illustrates video encoder 200 in accordance with one embodiment of the prior art.
  • Video encoder 200 comprises base layer encoding unit 210 and enhancement layer encoding unit 250 .
  • Video encoder 200 receives an original video signal that is transferred to base layer encoding unit 210 for generation of a base layer bit stream and to enhancement layer encoding unit 250 for generation of an enhancement layer bit stream.
  • Base layer encoding unit 210 contains a main processing branch, comprising motion estimator 212 , transform circuit 214 , quantization circuit 216 , entropy coder 218 , and buffer 220 , that generates the base layer bit stream.
  • Base layer encoding unit 210 comprises base layer rate allocator 222 , which is used to adjust the quantization factor of base layer encoding unit 210 .
  • Base layer encoding unit 210 also contains a feedback branch comprising inverse quantization circuit 224 , inverse transform circuit 226 , and frame store circuit 228 .
  • Motion estimator 212 receives the original video signal and estimates the amount of motion between a reference frame and the present video frame as represented by changes in pixel characteristics. For example, the MPEG standard specifies that motion information may be represented by one to four spatial motion vectors per 16 ⁇ 16 sub-block of the frame.
  • Transform circuit 214 receives the resultant motion difference estimate output from motion estimator 212 and transforms it from a spatial domain to a frequency domain using known de-correlation techniques, such as discrete cosine transform (DCT).
  • DCT discrete cosine transform
  • Quantization circuit 216 receives the DCT coefficient outputs from transform circuit 214 and a scaling factor from base layer rate allocator circuit 322 and further compresses the motion compensation prediction information using well-known quantization techniques. Quantization circuit 216 utilizes the scaling factor from base layer rate allocator circuit 222 to determine the division factor to be applied for quantization of the transform output.
  • entropy coder circuit 218 receives the quantized DCT coefficients from quantization circuit 216 and further compresses the data using variable length coding techniques that represent areas with a high probability of occurrence with a relatively short code and that represent areas of lower probability of occurrence with a relatively long code.
  • Buffer 220 receives the output of entropy coder 218 and provides necessary buffering for output of the compressed base layer bit stream. In addition, buffer 220 provides a feedback signal as a reference input for base layer rate allocator 222 . Base layer rate allocator 222 receives the feedback signal from buffer 220 and uses it in determining the division factor supplied to quantization circuit 216 .
  • Inverse quantization circuit 224 de-quantizes the output of quantization circuit 216 to produce a signal that is representative of the transform input to quantization circuit 216 .
  • Inverse transform circuit 226 decodes the output of inverse quantization circuit 224 to produce a signal which provides a frame representation of the original video signal as modified by the transform and quantization processes.
  • Frame store circuit 228 receives the decoded representative frame from inverse transform circuit 226 and stores the frame as a reference output to motion estimator circuit 212 and enhancement layer encoding unit 250 .
  • Motion estimator circuit 212 uses the resultant stored frame signal as the input reference signal for determining motion changes in the original video signal.
  • Enhancement layer encoding unit 250 contains a main processing branch, comprising residual calculator 252 , transform circuit 254 , and fine granular scalability (FGS) encoder 256 . Enhancement layer encoding unit 250 also comprises enhancement rate allocator 258 . Residual calculator circuit 252 receives frames from the original video signal and compares them with the decoded (or reconstructed) base layer frames in frame store 228 to produce a residual signal representing image information which is missing in the base layer frames as a result of the transform and quantization processes. The output of residual calculator circuit 252 is known as the residual data or residual error data.
  • Transform circuit 254 receives the output from residual calculator 252 and compresses this data using a known transform technique, such as DCT. Though DCT serves as the exemplary transform for this implementation, transform circuit 254 is not required to have the same transform process as base layer transform 214 .
  • FGS frame encoder circuit 256 receives outputs from transform circuit 254 and enhancement rate allocator 258 .
  • FGS frame encoder 256 encodes and compresses the DCT coefficients as adjusted by enhancement rate allocator 258 to produce the compressed output for the enhancement layer bit stream.
  • Enhancement rate allocator 258 receives the DCT coefficients from transform circuit 254 and utilizes them to produce a rate allocation control that is applied to FGS frame encoder circuit 256 .
  • the prior art implementation depicted in FIG. 2 results in an enhancement layer residual compressed signal that is representative of the difference between the original video signal and the decoded base layer data, with all residuals being processed without regard to the internal parameters of base layer encoding unit 310 .
  • the present invention uses one or more parameters taken from the base layer (e.g., motion vectors, base layer quantization errors, rate-control information, etc.) to improve the operation of the enhancement layer.
  • the new compression techniques may be applied independent of the transforms (e.g., discrete cosine transform (DCT) or wavelets) chosen in the base and enhancement layers.
  • DCT discrete cosine transform
  • FIG. 3 illustrates video encoder 114 in greater detail in accordance with one embodiment of the present invention.
  • video encoder 114 is similar to prior art video encoder 200 .
  • Video encoder 114 comprises base layer encoding unit 310 and enhancement layer encoding unit 350 .
  • Video encoder 114 receives an original video signal that is transferred to base layer encoding unit 310 for generation of a base layer bit stream and to enhancement layer encoding unit 350 for generation of an enhancement layer bit stream.
  • Base layer encoding unit 310 contains a main processing branch, comprising motion estimator 312 , transform circuit 314 , quantization circuit 316 , entropy coder 318 , and buffer 320 , that generates the base layer bit stream.
  • Base layer encoding unit 310 also comprises base layer rate allocator 322 , which is used to allocate the base layer data from base layer encoding unit 310 .
  • Base layer encoding unit 310 also contains a feedback branch comprising inverse quantization circuit 324 , inverse transform circuit 326 , and frame store circuit 328 .
  • Enhancement layer encoding unit 350 contains a main processing branch, comprising residual calculator 352 , transform circuit 354 , and fine granular scalability (FGS) encoder 356 . Enhancement layer encoding unit 350 also comprises enhancement layer rate allocator 358 .
  • enhancement layer encoding unit 350 in video encoder 114 also comprises base layer parameter monitor 300 .
  • the labels A, B, and C in base layer encoding unit 310 and enhancement layer encoding unit 350 represent signal lines that interconnect components in base layer encoding unit 310 with components in enhancement layer encoding unit 350 .
  • the signals lines are omitted for clarity.
  • exemplary base layer parameter monitor 300 may receive one or more base layer bit rate parameters from base layer rate allocator 322 .
  • label B indicates, exemplary base layer parameter monitor 300 also may receive one or more base layer quantization error parameters from quantization circuit 316 .
  • exemplary base layer parameter monitor 300 may receive one or more bases layer motion parameters from motion estimator 312 .
  • base layer parameter monitor 300 may also receive one or more base layer parameters from one or more of N other base layer encoders, arbitrarily labeled “High BL Rate Encoder 1” through “High BL Rate Encoder N,” that operate at higher rates that base layer rate allocator 322 .
  • Base layer parameter monitor 300 utilizes the base layer parameters from base layer encoding unit 310 and High BL Rate Encoders 1-N as reference signals to generate one or more output signals that control the operation of enhancement rate allocator 358 .
  • the output signals from base layer parameter monitor 300 adjust or modify the operation of enhancement rate allocator circuit 358 by re-allocating the way the enhancement layer data are distributed among block, groups of blocks, and frames in the base layer.
  • base layer parameter monitor 300 may also comprise comparator circuitry capable of receiving base layer frames from one or more of the N other base layer encoders that operate at higher rates that base layer encoding unit 310 and comparing them to the base layer frames in base layer encoding unit 310 .
  • Base layer parameter monitor 300 produces from these inputs a base layer difference signal that is used as a base layer parameter to adjust or modify the operation of enhancement rate allocator circuit 358 by re-allocating the way the enhancement layer data are distributed among block, groups of blocks, and frames in the base layer.
  • base layer parameters described above are exemplary only and not exhaustive. They should not be construed to exclude utilization of other base layer parameter signals suitable for improvements to enhancement layer video bit streams.
  • Enhancement rate allocator circuit 358 receives parameter output signal(s) from base layer parameter monitor 300 and an output from transform circuit 354 . Enhancement rate allocator 358 utilizes the received transform and parameter signals as the basis for developing the rate allocation output to FGS frame encoder circuit 356 .
  • Enhancement layer encoding unit 350 utilizes one or more of the base layer parameters and/or base layer rate encoder outputs to develop an improved enhancement layer bit stream that exploits the human eye's sensitivity to certain types of visual errors. For instance, enhancement layer encoding unit 350 may use base layer bit rate parameter as a guide for allocating additional bits within a particular video frame or between two or more video frames so that the image quality locally or globally approaches the quality of a system that is operating at a higher transmission rate. The resulting image provides a perceptually better image quality. A similar process may be used to insure more consistent image quality between consecutive frames.
  • enhancement layer encoding unit 350 may use the base layer quantization error parameter as the means for classifying residual errors introduced by the base layer quantization process.
  • FGS frame encoder circuit 356 then improves the resultant visual image by adding bits as compensation for the identified residual error class.
  • enhancement layer encoding unit 350 may minimize inter-block distortions introduced by the quantization process by assigning different transmission priorities to the various residual coefficients. Enhancement layer encoding unit 350 uses the resultant prioritized residual coefficients as the basis for reconstructing a perceptually more pleasing low bit rate image, which is characterized by smoother interblock transitions.
  • Enhancement layer encoding unit 350 may use the base layer motion parameter as the basis for classifying images by the degree of motion between frames, with the motion classification determining the amount of compression for particular images. Since discrepancies in images which are moving rapidly are less visible to the human eye, FGS frame encoder 356 may increase the compression of data representing rapidly moving images by decreasing the number of bits that represent the image. Conversely, FGS frame encoder 356 may allocate more bits for areas with slow or little motion, thus improving the visual perception of these images.
  • Base layer bit rate parameters received from High BL Rate Encoders 1-N provide an extension of the enhancement capability provided by the internal base layer bit rate parameter received from base layer encoding unit 310 .
  • Base layer parameter monitor 300 monitors the base layer bit rate parameters of High BL Rate Encoders 1-N and uses the information to provide an output representing the best compromise for image quality and rate.
  • Enhancement rate allocator 358 uses the output of base layer parameter monitor 300 to adjust the transform data received by FGS frame encoder 356 .
  • a specific embodiment may be realized by performing and recording additional single-layer encoding sessions at incremental bit-rates and subsequently reusing these reconstructed signals as guidelines for rate allocation at the enhancement layer.
  • the video is first encoded at the targeted base layer bit rate. Next, additional encoding sessions are performed at incremental bit rates, and the reconstructed signals for all the incremental encoding sessions are recorded.
  • the selected number of redundant single-layer encoding sessions should be a good trade-off between increased visual quality and consistency and the associated complexity.
  • the encoding is performed off-line and the encoder complexity is not a major constraint in the over-all system design, a larger number of redundant single-layer coding cycles could be tolerated.
  • an important component in the presented improvement is formed by the encoder/decoder synchronization.
  • the choices made at encoding time should be possible to reproduce at the decoder for adequate reconstruction. For example, if certain regions of the picture are coded differently (e.g., get a higher priority), the form and location of the particular region-of-interest should be transmitted to the decoder as side information. However, transmitting this side information (i.e., encoder choices) leads to an additional coding overhead. Therefore, an alternative is to define a mutual agreement between the encoder and decoder, based on, for example, base-layer information, such that the choices made by the enhancement encoder are similarly reproduced by the enhancement decoder without requiring additional side (synchronization) information transmission.
  • An example, where the encoder and decoder can be synchronized without the transmission of side information, can be based on the motion vectors already transmitted for the base-layer reconstruction (see above) or a particular object segmentation which can be performed on the decoded version of the base-layer picture, which the encoder and decoder both have available.
  • FIG. 4 is a flow diagram 400 illustrating the operation of exemplary video encoder 114 in accordance with one embodiment of the present invention.
  • Base layer parameter monitor 300 receives one or more base layer parameters from base layer encoding unit 310 or from High BL Rate Encoders 1-N (process step 405 ).
  • the base layer parameters may comprise one or more of a base layer bit rate parameter, a base layer quantization error parameter, a base layer motion parameter C, or other possible descriptive parameters.
  • Base layer parameter monitor 300 uses the base layer parameters to identify (or classify) one or more errors or poor image quality indicators, including visual masking factors, in the base layer (process step 410 ). Base layer parameter monitor 300 also determines the present allocation of enhancement layer data with respect to the base layer frames and blocks within the base layer frame (process step 415 ). Finally, base layer parameter monitor 300 controls enhancement rate allocator 358 in such as way as to modify the allocation of the enhancement layer data among the pixels blocks and frames of the base layer data. This results in a reduction or elimination of identified errors and/or poor image quality indicators (per step 420 ). The resultant output of FGS frame encoder 356 provides an enhancement layer bit stream which has been perceptually improved through the use of base layer parameter-based discrimination techniques.

Abstract

There is disclosed an apparatus for controlling the transmission of enhancement layer video data for use in a video encoder containing a base layer encoder and an enhancement layer encoder. The base layer encoder receives input video frames and generates compressed base layer video frames suitable for transmission at a base layer bit rate to a streaming video receiver. The enhancement layer encoder compares the input video frames and a processed version of the compressed base layer video frames and generates enhancement layer video data suitable for transmission at a modifiable enhancement layer bit rate to the streaming video receiver. The apparatus comprises a base layer parameter monitor for receiving at least one base layer parameter and, in response thereto, modifying an allocation of the enhancement layer video data among corresponding ones of the compressed base layer video frames.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
The present invention is related to that disclosed in United States patent application Ser. No. 09/347,882), entitled “SYSTEM AND METHOD FOR FINE GRANULAR SCALABLE VIDEO WITH SELECTIVE QUALITY ENHANCEMENT,” which is being filed concurrently herewith and is commonly assigned to the assignee of the present invention. The disclosure of the related patent application is incorporated herein by reference for all purposes as if fully set forth herein.
TECHNICAL FIELD OF THE INVENTION
The present invention is directed, in general, to video encoding systems and, more specifically, to an encoding system for streaming video data.
BACKGROUND OF THE INVENTION
Real-time streaming of multimedia content over data networks, including the Internet, has become an increasingly common application in recent years. A wide range of interactive and non-interactive multimedia applications, such as news-on-demand, live network television viewing, video conferencing, among others, rely on end-to-end streaming video techniques. Unlike a “downloaded” video file, which may be retrieved first in “non-real” time and viewed or played back later in “real” time, streaming video applications require a video transmitter that encodes and transmits a video signal over a data network to a video receiver, which must decode and display the video signal in real time.
Scalable video coding is a desirable feature for many multimedia applications and services that are used in systems employing decoders with a wide range of processing power. Scalability allows processors with low computational power to decode only a subset of the scalable video stream. Another use of scalable video is in environments with a variable transmission bandwidth. In those environments, receivers with low-access bandwidth receive, and consequently decode, only a subset of the scalable video stream, where the amount of that subset is proportional to the available bandwidth.
Several video scalability approaches have been adopted by lead video compression standards such as MPEG-2 and MPEG-4. Temporal, spatial, and quality (e.g., signal-noise ratio (SNR)) scalability types have been defined in these standards. All of these approaches consist of a base layer (BL) and an enhancement layer (EL). The base layer part of the scalable video stream represents, in general, the minimum amount of data needed for decoding that stream. The enhanced layer part of the stream represents additional information, and therefore enhances the video signal representation when decoded by the receiver.
For example, in a variable bandwidth system, such as the Internet, the base layer transmission rate may be established at the minimum guaranteed transmission rate of the variable bandwidth system. Hence, if a subscriber has a minimum guaranteed bandwidth of 256 kbps, the base layer rate may be established at 256 kbps also. If the actual available bandwidth is 384 kbps, the extra 128 kbps of bandwidth may be used by the enhancement layer to improve on the basic signal transmitted at the base layer rate.
For each type of video scalability, a certain scalability structure is identified. The scalability structure defines the relationship among the pictures of the base layer and the pictures of the enhanced layer. One class of scalability is fine-granular scalability. Images coded with this type of scalability can be decoded progressively. In other words, the decoder may decode and display the image with only a subset of the data used for coding that image. As more data is received, the quality of the decoded image is progressively enhanced until the complete information is received, decoded, and displayed.
The newly proposed MPEG-4 standard is directed to new video streaming applications based on very low bit rate coding, such as video-phone, mobile multimedia and audio-visual communications, multimedia e-mail, remote sensing, interactive games, and the like. Within the MPEG-4 standard, fine-granular scalability (FGS) has been recognized as an essential technique for networked video distribution. FGS primarily targets applications where video is streamed over heterogeneous networks in real-time. It provides bandwidth adaptivity by encoding content once for a range of bit rates, and enabling the video transmission server to change the transmission rate dynamically without in-depth knowledge or parsing of the video bit stream.
An important priority within conventional FGS techniques is improving coding efficiency and visual quality of the intra-frame coded enhancement layer. This is necessary to justify the adoption of FGS techniques for the compression of the enhancement layer in place of non-scalable (e.g., single layer) or less granular (e.g., multi-level SNR scalability) coding methods.
A limitation of the compression scheme currently adopted as reference for FGS resides in its inability to exploit the base layer coding information for improving the compression efficiency of the enhancement-layer. Another disadvantage of currently adopted FGS schemes resides in the fact that enhancement layer frames are coded independently of each other (i.e., “intra” coding of frames). The intra-frame coding of the enhancement layer is necessary for error resilience and for easy bit rate change at transmission time. However, because each enhancement frame is optimally coded in its own context, discontinuity or inconsistency between the image quality of consecutive frames is often introduced. The resulting FGS enhanced video may have “flashing” artifacts across frames. This is particular annoying and highly visible when compared to the more “visually stable” single layer coded video.
There is therefore a need in the art for improved encoders and encoding techniques for use in streaming video systems. There is a further need for encoders and encoding techniques that are less susceptible to flashing artifacts and other sources of discontinuity in the quality of consecutive frames in a sequence of related frames. In particular there is a need in the art for encoders that selectively allocate the enhancement layer data in relation to the amount of activity or selected characteristics in the original video image.
SUMMARY OF THE INVENTION
To address the above-discussed deficiencies of the prior art, it is a primary object of the present invention to provide a new technique for improving the coding efficiency of the enhancement layer compression scheme. The proposed encoding technique uses one or more parameters taken from the base layer compression information (e.g., motion vectors, base layer quantization errors, rate-control info, etc.) to improve the image quality of the enhancement layer. Moreover, based on the observation that single layer encoding usually does a good job in optimizing video quality for particular bit rates, the present invention may use single layer coding at multiple bit rates as “guidelines” for FGS encoding. The new compression techniques may be applied independent of the transforms chosen in the base and enhancement layers (e.g., discrete cosine transform (DCT) or wavelets). However, the use of certain base layer or single-layer information is less straightforward if different coding schemes are employed at the base and enhancement layers.
Accordingly, in an advantageous embodiment of the present invention, there is provided, for use in a video encoder comprising a base layer circuit capable of receiving an input stream of video frames and generating therefrom compressed base layer video frames suitable for transmission at a base layer bit rate to a streaming video receiver and an enhancement layer circuit capable of receiving the input stream of video frames and a decoded version of the compressed base layer video frames and generating therefrom enhancement layer video data associated with, and allocated to, corresponding ones of the compressed base layer video frames and suitable for transmission at a modifiable enhancement layer bit rate to the streaming video receiver an apparatus for controlling transmission of the enhancement layer video data. The apparatus comprises a base layer parameter monitor capable of receiving at least one base layer parameter and, in response thereto, modifying an allocation of the enhancement layer video data among the corresponding ones of the compressed base layer video frames.
In one embodiment of the present invention, the video encoder comprises a motion estimation circuit capable of receiving the input stream of video frames and determining therefrom a base layer motion parameter associated with at least one selected frame sequence in the input stream of video frames.
In another embodiment of the present invention, the base layer parameter monitor receives the base layer motion parameter and, in response thereto, modifies the allocation of the enhancement layer video data according to a level of motion in the at least one selected frame sequence indicated by the base layer motion parameter.
In still another embodiment of the present invention, the video encoder comprises a quantization circuit capable of receiving and quantizing transform data associated with the input stream of video frames to thereby reduce a size of the transform data and further capable of determining a base layer quantization error parameter associated with the quantized transform data.
In a further embodiment of the present invention, the base layer parameter monitor receives the base layer quantization error parameter and, in response thereto, modifies the allocation of the enhancement layer video data according to a quantization error indicated by the base layer quantization error parameter.
In a still further embodiment of the present invention, the video encoder comprises a base layer rate allocation circuit capable of determining the base layer bit rate, wherein the base layer bit rate is set at a pre-determined minimum rate at which the compressed base layer video frames are transmitted to the streaming video receiver, and generating therefrom a base layer bit rate parameter associated with the base layer bit rate.
In a yet further embodiment of the present invention, the base layer parameter monitor receives the base layer bit rate parameter and, in response thereto, modifies the allocation of the enhancement layer video data according to an estimated difference between the compressed base layer video frames and estimated compressed base layer video frames associated with a second base layer bit rate greater than the pre-determined minimum rate.
The foregoing has outlined rather broadly the features and technical advantages of the present invention so that those skilled in the art may better understand the detailed description of the invention that follows. Additional features and advantages of the invention will be described hereinafter that form the subject of the claims of the invention. Those skilled in the art should appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the invention in its broadest form.
Before undertaking the Detailed Description, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or,” is inclusive, meaning and/or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.
BRIEF DESCRIPTION OF THE DRAWINGS
For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like numbers designate like objects, and in which:
FIG. 1 illustrates an end-to-end transmission of streaming video from a streaming video transmitter through a data network to a streaming video receiver, according to one embodiment of the present invention;
FIG. 2 illustrates a video encoder in accordance with one embodiment of the prior art;
FIG. 3 illustrates an exemplary video encoder in accordance with one embodiment of the present invention; and
FIG. 4 is a flow diagram illustrating the operation of an exemplary video encoder in accordance with one embodiment of the present invention.
DETAILED DESCRIPTION
FIGS. 1 through 4, discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the present invention may be implemented in any suitably arranged video encoder.
FIG. 1 illustrates an end-to-end transmission of streaming video from streaming video transmitter 110 through data network 120 to streaming video receiver 130, according to one embodiment of the present invention. Depending on the application, streaming video transmitter 110 may be any one of a wide variety of sources of video frames, including a data network server, a television station, a cable network, a desktop personal computer (PC), or the like.
Streaming video transmitter 110 comprises video frame source 112, video encoder 114 and recoder buffer 116. Video frame source 112 may be any device capable of generating a sequence of uncompressed video frames, including a television antenna and receiver unit, a video cassette player, a video camera, a disk storage device capable of storing a “raw” video clip, and the like. The uncompressed video frames enter video encoder 114 at a given picture rate (or “streaming rate”) and are compressed according to any known compression algorithm or device, such as an MPEG-4 encoder. Video encoder 114 then transmits the compressed video frames to encoder buffer 116 for buffering in preparation for transmission across data network 120. Data network 120 may be any suitable network and may include portions of both public data networks, such as the Internet, and private data networks, such as an enterprise-owned local area network (LAN) or wide area network (WAN).
Streaming video receiver 130 comprises decoder buffer 132, video decoder 134 and video display 136. Decoder buffer 132 receives and stores streaming compressed video frames from data network 120. Decoder buffer 132 then transmits the compressed video frames to video decoder 134 as required. Video decoder 134 decompresses the video frames at the same rate (ideally) at which the video frames were compressed by video encoder 114. Video decoder 134 sends the decompressed frames to video display 136 for play-back on the screen of video display 134.
FIG. 2 illustrates video encoder 200 in accordance with one embodiment of the prior art. Video encoder 200 comprises base layer encoding unit 210 and enhancement layer encoding unit 250. Video encoder 200 receives an original video signal that is transferred to base layer encoding unit 210 for generation of a base layer bit stream and to enhancement layer encoding unit 250 for generation of an enhancement layer bit stream.
Base layer encoding unit 210 contains a main processing branch, comprising motion estimator 212, transform circuit 214, quantization circuit 216, entropy coder 218, and buffer 220, that generates the base layer bit stream. Base layer encoding unit 210 comprises base layer rate allocator 222, which is used to adjust the quantization factor of base layer encoding unit 210. Base layer encoding unit 210 also contains a feedback branch comprising inverse quantization circuit 224, inverse transform circuit 226, and frame store circuit 228.
Motion estimator 212 receives the original video signal and estimates the amount of motion between a reference frame and the present video frame as represented by changes in pixel characteristics. For example, the MPEG standard specifies that motion information may be represented by one to four spatial motion vectors per 16×16 sub-block of the frame. Transform circuit 214 receives the resultant motion difference estimate output from motion estimator 212 and transforms it from a spatial domain to a frequency domain using known de-correlation techniques, such as discrete cosine transform (DCT).
Quantization circuit 216 receives the DCT coefficient outputs from transform circuit 214 and a scaling factor from base layer rate allocator circuit 322 and further compresses the motion compensation prediction information using well-known quantization techniques. Quantization circuit 216 utilizes the scaling factor from base layer rate allocator circuit 222 to determine the division factor to be applied for quantization of the transform output. Next, entropy coder circuit 218 receives the quantized DCT coefficients from quantization circuit 216 and further compresses the data using variable length coding techniques that represent areas with a high probability of occurrence with a relatively short code and that represent areas of lower probability of occurrence with a relatively long code.
Buffer 220 receives the output of entropy coder 218 and provides necessary buffering for output of the compressed base layer bit stream. In addition, buffer 220 provides a feedback signal as a reference input for base layer rate allocator 222. Base layer rate allocator 222 receives the feedback signal from buffer 220 and uses it in determining the division factor supplied to quantization circuit 216.
Inverse quantization circuit 224 de-quantizes the output of quantization circuit 216 to produce a signal that is representative of the transform input to quantization circuit 216. Inverse transform circuit 226 decodes the output of inverse quantization circuit 224 to produce a signal which provides a frame representation of the original video signal as modified by the transform and quantization processes. Frame store circuit 228 receives the decoded representative frame from inverse transform circuit 226 and stores the frame as a reference output to motion estimator circuit 212 and enhancement layer encoding unit 250. Motion estimator circuit 212 uses the resultant stored frame signal as the input reference signal for determining motion changes in the original video signal.
Enhancement layer encoding unit 250 contains a main processing branch, comprising residual calculator 252, transform circuit 254, and fine granular scalability (FGS) encoder 256. Enhancement layer encoding unit 250 also comprises enhancement rate allocator 258. Residual calculator circuit 252 receives frames from the original video signal and compares them with the decoded (or reconstructed) base layer frames in frame store 228 to produce a residual signal representing image information which is missing in the base layer frames as a result of the transform and quantization processes. The output of residual calculator circuit 252 is known as the residual data or residual error data.
Transform circuit 254 receives the output from residual calculator 252 and compresses this data using a known transform technique, such as DCT. Though DCT serves as the exemplary transform for this implementation, transform circuit 254 is not required to have the same transform process as base layer transform 214.
FGS frame encoder circuit 256 receives outputs from transform circuit 254 and enhancement rate allocator 258. FGS frame encoder 256 encodes and compresses the DCT coefficients as adjusted by enhancement rate allocator 258 to produce the compressed output for the enhancement layer bit stream. Enhancement rate allocator 258 receives the DCT coefficients from transform circuit 254 and utilizes them to produce a rate allocation control that is applied to FGS frame encoder circuit 256.
The prior art implementation depicted in FIG. 2 results in an enhancement layer residual compressed signal that is representative of the difference between the original video signal and the decoded base layer data, with all residuals being processed without regard to the internal parameters of base layer encoding unit 310. The present invention, as described below, uses one or more parameters taken from the base layer (e.g., motion vectors, base layer quantization errors, rate-control information, etc.) to improve the operation of the enhancement layer. The new compression techniques may be applied independent of the transforms (e.g., discrete cosine transform (DCT) or wavelets) chosen in the base and enhancement layers.
FIG. 3 illustrates video encoder 114 in greater detail in accordance with one embodiment of the present invention. For the most part, video encoder 114 is similar to prior art video encoder 200. Video encoder 114 comprises base layer encoding unit 310 and enhancement layer encoding unit 350. Video encoder 114 receives an original video signal that is transferred to base layer encoding unit 310 for generation of a base layer bit stream and to enhancement layer encoding unit 350 for generation of an enhancement layer bit stream.
Base layer encoding unit 310 contains a main processing branch, comprising motion estimator 312, transform circuit 314, quantization circuit 316, entropy coder 318, and buffer 320, that generates the base layer bit stream. Base layer encoding unit 310 also comprises base layer rate allocator 322, which is used to allocate the base layer data from base layer encoding unit 310. Base layer encoding unit 310 also contains a feedback branch comprising inverse quantization circuit 324, inverse transform circuit 326, and frame store circuit 328.
Enhancement layer encoding unit 350 contains a main processing branch, comprising residual calculator 352, transform circuit 354, and fine granular scalability (FGS) encoder 356. Enhancement layer encoding unit 350 also comprises enhancement layer rate allocator 358.
However, unlike prior art video encoder 200, enhancement layer encoding unit 350 in video encoder 114 also comprises base layer parameter monitor 300. The labels A, B, and C in base layer encoding unit 310 and enhancement layer encoding unit 350 represent signal lines that interconnect components in base layer encoding unit 310 with components in enhancement layer encoding unit 350. The signals lines are omitted for clarity. As label A indicates, exemplary base layer parameter monitor 300 may receive one or more base layer bit rate parameters from base layer rate allocator 322. As label B indicates, exemplary base layer parameter monitor 300 also may receive one or more base layer quantization error parameters from quantization circuit 316. Finally, as label C indicates, exemplary base layer parameter monitor 300 may receive one or more bases layer motion parameters from motion estimator 312.
In a server environment containing multiple encoders operating a different base layer rates, base layer parameter monitor 300 may also receive one or more base layer parameters from one or more of N other base layer encoders, arbitrarily labeled “High BL Rate Encoder 1” through “High BL Rate Encoder N,” that operate at higher rates that base layer rate allocator 322.
Base layer parameter monitor 300 utilizes the base layer parameters from base layer encoding unit 310 and High BL Rate Encoders 1-N as reference signals to generate one or more output signals that control the operation of enhancement rate allocator 358. The output signals from base layer parameter monitor 300 adjust or modify the operation of enhancement rate allocator circuit 358 by re-allocating the way the enhancement layer data are distributed among block, groups of blocks, and frames in the base layer.
In an alternate embodiment of the present invention, base layer parameter monitor 300 may also comprise comparator circuitry capable of receiving base layer frames from one or more of the N other base layer encoders that operate at higher rates that base layer encoding unit 310 and comparing them to the base layer frames in base layer encoding unit 310. Base layer parameter monitor 300 produces from these inputs a base layer difference signal that is used as a base layer parameter to adjust or modify the operation of enhancement rate allocator circuit 358 by re-allocating the way the enhancement layer data are distributed among block, groups of blocks, and frames in the base layer.
It should be noted that the base layer parameters described above are exemplary only and not exhaustive. They should not be construed to exclude utilization of other base layer parameter signals suitable for improvements to enhancement layer video bit streams.
Enhancement rate allocator circuit 358 receives parameter output signal(s) from base layer parameter monitor 300 and an output from transform circuit 354. Enhancement rate allocator 358 utilizes the received transform and parameter signals as the basis for developing the rate allocation output to FGS frame encoder circuit 356.
Enhancement layer encoding unit 350 utilizes one or more of the base layer parameters and/or base layer rate encoder outputs to develop an improved enhancement layer bit stream that exploits the human eye's sensitivity to certain types of visual errors. For instance, enhancement layer encoding unit 350 may use base layer bit rate parameter as a guide for allocating additional bits within a particular video frame or between two or more video frames so that the image quality locally or globally approaches the quality of a system that is operating at a higher transmission rate. The resulting image provides a perceptually better image quality. A similar process may be used to insure more consistent image quality between consecutive frames.
In a similar manner, enhancement layer encoding unit 350 may use the base layer quantization error parameter as the means for classifying residual errors introduced by the base layer quantization process. FGS frame encoder circuit 356 then improves the resultant visual image by adding bits as compensation for the identified residual error class. Moreover, enhancement layer encoding unit 350 may minimize inter-block distortions introduced by the quantization process by assigning different transmission priorities to the various residual coefficients. Enhancement layer encoding unit 350 uses the resultant prioritized residual coefficients as the basis for reconstructing a perceptually more pleasing low bit rate image, which is characterized by smoother interblock transitions.
Enhancement layer encoding unit 350 may use the base layer motion parameter as the basis for classifying images by the degree of motion between frames, with the motion classification determining the amount of compression for particular images. Since discrepancies in images which are moving rapidly are less visible to the human eye, FGS frame encoder 356 may increase the compression of data representing rapidly moving images by decreasing the number of bits that represent the image. Conversely, FGS frame encoder 356 may allocate more bits for areas with slow or little motion, thus improving the visual perception of these images.
Base layer bit rate parameters received from High BL Rate Encoders 1-N provide an extension of the enhancement capability provided by the internal base layer bit rate parameter received from base layer encoding unit 310. Base layer parameter monitor 300 monitors the base layer bit rate parameters of High BL Rate Encoders 1-N and uses the information to provide an output representing the best compromise for image quality and rate. Enhancement rate allocator 358 uses the output of base layer parameter monitor 300 to adjust the transform data received by FGS frame encoder 356.
A specific embodiment may be realized by performing and recording additional single-layer encoding sessions at incremental bit-rates and subsequently reusing these reconstructed signals as guidelines for rate allocation at the enhancement layer. The video is first encoded at the targeted base layer bit rate. Next, additional encoding sessions are performed at incremental bit rates, and the reconstructed signals for all the incremental encoding sessions are recorded.
The selected number of redundant single-layer encoding sessions should be a good trade-off between increased visual quality and consistency and the associated complexity. However, in the case the encoding is performed off-line and the encoder complexity is not a major constraint in the over-all system design, a larger number of redundant single-layer coding cycles could be tolerated.
An important component in the presented improvement is formed by the encoder/decoder synchronization. The choices made at encoding time should be possible to reproduce at the decoder for adequate reconstruction. For example, if certain regions of the picture are coded differently (e.g., get a higher priority), the form and location of the particular region-of-interest should be transmitted to the decoder as side information. However, transmitting this side information (i.e., encoder choices) leads to an additional coding overhead. Therefore, an alternative is to define a mutual agreement between the encoder and decoder, based on, for example, base-layer information, such that the choices made by the enhancement encoder are similarly reproduced by the enhancement decoder without requiring additional side (synchronization) information transmission. An example, where the encoder and decoder can be synchronized without the transmission of side information, can be based on the motion vectors already transmitted for the base-layer reconstruction (see above) or a particular object segmentation which can be performed on the decoded version of the base-layer picture, which the encoder and decoder both have available.
FIG. 4 is a flow diagram 400 illustrating the operation of exemplary video encoder 114 in accordance with one embodiment of the present invention. Base layer parameter monitor 300 receives one or more base layer parameters from base layer encoding unit 310 or from High BL Rate Encoders 1-N (process step 405). The base layer parameters may comprise one or more of a base layer bit rate parameter, a base layer quantization error parameter, a base layer motion parameter C, or other possible descriptive parameters.
Base layer parameter monitor 300 uses the base layer parameters to identify (or classify) one or more errors or poor image quality indicators, including visual masking factors, in the base layer (process step 410). Base layer parameter monitor 300 also determines the present allocation of enhancement layer data with respect to the base layer frames and blocks within the base layer frame (process step 415). Finally, base layer parameter monitor 300 controls enhancement rate allocator 358 in such as way as to modify the allocation of the enhancement layer data among the pixels blocks and frames of the base layer data. This results in a reduction or elimination of identified errors and/or poor image quality indicators (per step 420). The resultant output of FGS frame encoder 356 provides an enhancement layer bit stream which has been perceptually improved through the use of base layer parameter-based discrimination techniques.
Although the present invention has been described in detail, those skilled in the art should understand that they can make various changes, substitutions and alterations herein without departing from the spirit and scope of the invention in its broadest form.

Claims (21)

What is claimed is:
1. For use in a video encoder comprising: 1) a base layer circuit capable of receiving an input stream of video frames and generating therefrom compressed base layer video frames suitable for transmission at a base layer bit rate to a streaming video receiver, and 2) an enhancement layer circuit capable of receiving said input stream of video frames and a decoded version of said compressed base layer video frames and generating therefrom enhancement layer video data associated with, and allocated to, corresponding ones of said compressed base layer video frames and suitable for transmission at a modifiable enhancement layer bit rate to said streaming video receiver, an apparatus for controlling transmission of said enhancement layer video data comprising:
a base layer parameter monitor capable of receiving at least one base layer parameter and, in response thereto, developing a rate allocation output and modifying an allocation of said enhancement layer video data among said corresponding ones of said compressed base layer video frames.
2. The apparatus set forth in claim 1 wherein said video encoder comprises a motion estimation circuit capable of receiving said input stream of video frames and determining therefrom a base layer motion parameter associated with at least one selected frame sequence in said input stream of video frames.
3. The apparatus set forth in claim 2 wherein said base layer parameter monitor receives said base layer motion parameter and, in response thereto, modifies said allocation of said enhancement layer video data according to a level of motion in said at least one selected frame sequence indicated by said base layer motion parameter.
4. The apparatus set forth in claim 1 wherein said video encoder comprises a quantization circuit capable of receiving and quantizing transform data associated with said input stream of video frames to thereby reduce a size of said transform data and further capable of determining a base layer quantization error parameter associated with said quantized transform data.
5. The apparatus set forth in claim 4 wherein said base layer parameter monitor receives said base layer quantization error parameter and, in response thereto, modifies said allocation of said enhancement layer video data according to a quantization error indicated by said base layer quantization error parameter.
6. The apparatus set forth in claim 1 wherein said video encoder comprises a base layer rate allocation circuit capable of determining said base layer bit rate, wherein said base layer bit rate is set at a pre-determined minimum rate at which said compressed base layer video frames are transmitted to said streaming video receiver, and generating therefrom a base layer bit rate parameter associated with said base layer bit rate.
7. The apparatus set forth in claim 6 wherein said base layer parameter monitor receives said base layer bit rate parameter and, in response thereto, modifies said allocation of said enhancement layer video data according to an estimated difference between said compressed base layer video frames and estimated compressed base layer video frames associated with a second base layer bit rate greater than said pre-determined minimum rate.
8. For use in a data network comprising a plurality of nodes capable of receiving streaming video data, a streaming video transmitter capable of transmitting said streaming video data to one or more of said nodes, said streaming video transmitter comprising:
a video frame source capable of generating an original stream of video frames; and
a video encoder comprising:
a base layer circuit capable of receiving said original stream of video frames and generating therefrom compressed base layer video frames suitable for transmission at a base layer bit rate to said one or more of said nodes;
an enhancement layer circuit capable of receiving said original stream of video frames and a decoded version of said compressed base layer video frames, developing a rate allocation output and generating therefrom enhancement layer video data associated with, and allocated to, corresponding ones of said compressed base layer video frames and suitable for transmission at a modifiable enhancement layer bit rate to said one or more of said nodes; and
an apparatus for controlling transmission of said enhancement layer video data comprising a base layer parameter monitor capable of receiving at least one base layer parameter and, in response thereto, developing a rate allocation output and modifying an allocation of said enhancement layer video data among said corresponding ones of said compressed base layer video frames.
9. The streaming video transmitter set forth in claim 8 wherein said video encoder comprises a motion estimation circuit capable of receiving said input stream of video frames and determining therefrom a base layer motion parameter associated with at least one selected frame sequence in said input stream of video frames.
10. The streaming video transmitter set forth in claim 9 wherein said base layer parameter monitor receives said base layer motion parameter and, in response thereto, modifies said allocation of said enhancement layer video data according to a level of motion in said at least one selected frame sequence indicated by said base layer motion parameter.
11. The streaming video transmitter set forth in claim 8 wherein said video encoder comprises a quantization circuit capable of receiving and quantizing transform data associated with said input stream of video frames to thereby reduce a size of said transform data and further capable of determining a base layer quantization error parameter associated with said quantized transform data.
12. The streaming video transmitter set forth in claim 11 wherein said base layer parameter monitor receives said base layer quantization error parameter and, in response thereto, modifies said allocation of said enhancement layer video data according to a quantization error indicated by said base layer quantization error parameter.
13. The streaming video transmitter set forth in claim 8 wherein said video encoder comprises a base layer rate allocation circuit capable of determining said base layer bit rate, wherein said base layer bit rate is set at a pre-determined minimum rate at which said compressed base layer video frames are transmitted to said streaming video receiver, and generating therefrom a base layer bit rate parameter associated with said base layer bit rate.
14. The streaming video transmitter set forth in claim 13 wherein said base layer parameter monitor receives said base layer bit rate parameter and, in response thereto, modifies said allocation of said enhancement layer video data according to an estimated difference between said compressed base layer video frames and estimated compressed base layer video frames associated with a second base layer bit rate greater than said pre-determined minimum rate.
15. For use in a video encoder comprising: 1) a base layer circuit capable of receiving an input stream of video frames and generating therefrom compressed base layer video frames suitable for transmission at a base layer bit rate to a streaming video receiver, and 2) an enhancement layer circuit capable of receiving the input stream of video frames and a decoded version of the compressed base layer video frames and generating therefrom enhancement layer video data associated with, and allocated to, corresponding ones of the compressed base layer video frames and suitable for transmission at a modifiable enhancement layer bit rate to the streaming video receiver, a method for controlling a transmission of the enhancement layer video data comprising the steps of:
monitoring at least one base layer parameter; and
in response to a value of the monitored at least one base layer parameter, developing a rate allocation output and modifying an allocation of the enhancement layer video data among the corresponding ones of the compressed base layer video frames.
16. The method set forth in claim 15 wherein the video encoder comprises a motion estimation circuit capable of receiving the input stream of video frames and determining therefrom a base layer motion parameter associated with at least one selected frame sequence in the input stream of video frames.
17. The method set forth in claim 16 further comprising the steps of monitoring the base layer motion parameter and, in response thereto, modifying the allocation of the enhancement layer video data according to a level of motion in the at least one selected frame sequence indicated by the base layer motion parameter.
18. The method set forth in claim 15 wherein the video encoder comprises a quantization circuit capable of receiving and quantizing transform data associated with the input stream of video frames to thereby reduce a size of the transform data and further capable of determining a base layer quantization error parameter associated with the quantized transform data.
19. The method set forth in claim 18 further comprising the steps of monitoring the base layer quantization error parameter and, in response thereto, modifying the allocation of the enhancement layer video data according to a quantization error indicated by the base layer quantization error parameter.
20. The method set forth in claim 15 wherein the video encoder comprises a base layer rate allocation circuit capable of determining the base layer bit rate, wherein the base layer bit rate is set at a pre-determined minimum rate at which the compressed base layer video frames are transmitted to the streaming video receiver, and generating therefrom a base layer bit rate parameter associated with the base layer bit rate.
21. The method set forth in claim 20 further comprising the steps of monitoring the base layer bit rate parameter and, in response thereto, modifying the allocation of the enhancement layer video data according to an estimated difference between the compressed base layer video frames and estimated compressed base layer video frames associated with a second base layer bit rate greater than the pre-determined minimum rate.
US09/347,881 1999-07-06 1999-07-06 System and method for improved fine granular scalable video using base layer coding information Expired - Lifetime US6501797B1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
US09/347,881 US6501797B1 (en) 1999-07-06 1999-07-06 System and method for improved fine granular scalable video using base layer coding information
AU65609/00A AU6560900A (en) 1999-07-06 2000-07-03 System and method for improved fine granular scalable video using base layer coding information
PCT/EP2000/006243 WO2001003441A1 (en) 1999-07-06 2000-07-03 System and method for improved fine granular scalable video using base layer coding information
EP00952999A EP1110405A1 (en) 1999-07-06 2000-07-03 System and method for improved fine granular scalable video using base layer coding information
CNB00801843XA CN1192629C (en) 1999-07-06 2000-07-03 System and method for improved fine granular scalable video using base layer coding information
KR1020017002917A KR20010086365A (en) 1999-07-06 2000-07-03 System and method for improved fine granular scalable video using base layer coding information
JP2001508171A JP2003533067A (en) 1999-07-06 2000-07-03 System and method for improved definition scalable video by using reference layer coded information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US09/347,881 US6501797B1 (en) 1999-07-06 1999-07-06 System and method for improved fine granular scalable video using base layer coding information

Publications (1)

Publication Number Publication Date
US6501797B1 true US6501797B1 (en) 2002-12-31

Family

ID=23365685

Family Applications (1)

Application Number Title Priority Date Filing Date
US09/347,881 Expired - Lifetime US6501797B1 (en) 1999-07-06 1999-07-06 System and method for improved fine granular scalable video using base layer coding information

Country Status (7)

Country Link
US (1) US6501797B1 (en)
EP (1) EP1110405A1 (en)
JP (1) JP2003533067A (en)
KR (1) KR20010086365A (en)
CN (1) CN1192629C (en)
AU (1) AU6560900A (en)
WO (1) WO2001003441A1 (en)

Cited By (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020080878A1 (en) * 2000-10-12 2002-06-27 Webcast Technologies, Inc. Video apparatus and method for digital video enhancement
US20020118743A1 (en) * 2001-02-28 2002-08-29 Hong Jiang Method, apparatus and system for multiple-layer scalable video coding
US20030035480A1 (en) * 2001-08-15 2003-02-20 Philips Electronics North America Corporation Method for transmission control in hybrid temporal-SNR fine granular video coding
US6639943B1 (en) * 1999-11-23 2003-10-28 Koninklijke Philips Electronics N.V. Hybrid temporal-SNR fine granular scalability video coding
US20040071083A1 (en) * 2002-02-22 2004-04-15 Koninklijke Philips Electronics N.V. Method for streaming fine granular scalability coded video over an IP network
US20040103216A1 (en) * 2002-11-26 2004-05-27 Lane Richard D. System and method for optimizing multimedia compression using plural encoders
WO2004068861A1 (en) * 2003-01-30 2004-08-12 Koninklijke Philips Electronics N.V. Video coding
US6798838B1 (en) * 2000-03-02 2004-09-28 Koninklijke Philips Electronics N.V. System and method for improving video transmission over a wireless network
WO2004084520A1 (en) * 2003-03-19 2004-09-30 British Telecommunications Public Limited Company Data transmission over a network having initiallly undetermined transmission capacity
US20040261113A1 (en) * 2001-06-18 2004-12-23 Baldine-Brunel Paul Method of transmitting layered video-coded information
US20050135476A1 (en) * 2002-01-30 2005-06-23 Philippe Gentric Streaming multimedia data over a network having a variable bandwith
US20050157794A1 (en) * 2004-01-16 2005-07-21 Samsung Electronics Co., Ltd. Scalable video encoding method and apparatus supporting closed-loop optimization
US20050169386A1 (en) * 2004-01-13 2005-08-04 Gerd Spalink Method for pre-processing digital data, digital to analog and analog to digital conversion system
US20050180646A1 (en) * 2004-02-09 2005-08-18 Canon Kabushiki Kaisha Methods for sending and receiving an animation, and associated devices
US20060013309A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Video encoding and decoding methods and video encoder and decoder
WO2006006835A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Scalable motion information encoding/decoding apparatus and method and scalable video encoding/decoding apparatus and method using them
WO2006006793A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Video encoding and decoding methods and video encoder and decoder
US20060133514A1 (en) * 2002-03-27 2006-06-22 Walker Matthew D Video coding and transmission
US20060233250A1 (en) * 2005-04-13 2006-10-19 Samsung Electronics Co., Ltd. Method and apparatus for encoding and decoding video signals in intra-base-layer prediction mode by selectively applying intra-coding
WO2006108917A1 (en) * 2005-04-13 2006-10-19 Nokia Corporation Coding, storage and signalling of scalability information
WO2007011160A1 (en) * 2005-07-19 2007-01-25 Electronics And Telecommunications Research Institute Apparatus and method of embedded quantizaton for the improved snr scalbilty
US20070127585A1 (en) * 2005-12-06 2007-06-07 Fujitsu Limited Encoding apparatus, encoding method, and computer product
US7269785B1 (en) * 1999-12-30 2007-09-11 Genesis Microchip Inc. Digital manipulation of video in digital video player
US20080152003A1 (en) * 2006-12-22 2008-06-26 Qualcomm Incorporated Multimedia data reorganization between base layer and enhancement layer
US20080193033A1 (en) * 2005-07-19 2008-08-14 Hae Chul Choi Apparatus and Method of Embedded Quantization for the Improved Snr Scalbility
US20090175358A1 (en) * 2008-01-03 2009-07-09 Broadcom Corporation Video processing system and transcoder for use with layered video coding and methods for use therewith
US20090279605A1 (en) * 2008-05-07 2009-11-12 Microsoft Corporation Encoding streaming media as a high bit rate layer, a low bit rate layer, and one or more intermediate bit rate layers
US20090316835A1 (en) * 2005-03-31 2009-12-24 Qualcomm Incorporated Power savings in hierarchically coded modulation
US20100149301A1 (en) * 2008-12-15 2010-06-17 Microsoft Corporation Video Conferencing Subscription Using Multiple Bit Rate Streams
US20100153574A1 (en) * 2008-12-15 2010-06-17 Microsoft Corporation Video Conference Rate Matching
US7848433B1 (en) * 2000-11-22 2010-12-07 At&T Intellectual Property Ii, L.P. System and method for processing data with drift control
US7974200B2 (en) 2000-11-29 2011-07-05 British Telecommunications Public Limited Company Transmitting and receiving real-time data
US20110310216A1 (en) * 2010-06-18 2011-12-22 Microsoft Corporation Combining multiple bit rate and scalable video coding
US20110316965A1 (en) * 2010-06-25 2011-12-29 Microsoft Corporation Combining direct and routed communication in a video conference
US8135852B2 (en) 2002-03-27 2012-03-13 British Telecommunications Public Limited Company Data streaming system and method
US20120183045A1 (en) * 2011-01-18 2012-07-19 Louis Joseph Kerofsky Video decoder with reduced dynamic range transform including clipping
US20120183046A1 (en) * 2011-01-18 2012-07-19 Louis Joseph Kerofsky Video decoder with reduced dynamic range transform with inverse transform shifting memory
US8265140B2 (en) 2008-09-30 2012-09-11 Microsoft Corporation Fine-grained client-side control of scalable media delivery
US8370887B2 (en) 2008-05-30 2013-02-05 Microsoft Corporation Media streaming with enhanced seek operation
US8379851B2 (en) 2008-05-12 2013-02-19 Microsoft Corporation Optimized client side rate control and indexed file layout for streaming media
US20130195198A1 (en) * 2012-01-23 2013-08-01 Splashtop Inc. Remote protocol
US20150020131A1 (en) * 2012-01-20 2015-01-15 Korea Electronics Technology Institute Method for transmitting and receiving program configuration information for scalable ultra high definition video service in hybrid transmission environment, and method and apparatus for effectively transmitting scalar layer information
US20150036753A1 (en) * 2012-03-30 2015-02-05 Sony Corporation Image processing device and method, and recording medium
US20150103887A1 (en) * 2013-10-14 2015-04-16 Qualcomm Incorporated Device and method for scalable coding of video information
TWI488492B (en) * 2007-04-18 2015-06-11 Thomson Licensing Decoding apparatus
US20150172680A1 (en) * 2013-12-16 2015-06-18 Arris Enterprises, Inc. Producing an Output Need Parameter for an Encoder
US9171577B1 (en) * 2003-04-25 2015-10-27 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US9305590B2 (en) 2007-10-16 2016-04-05 Seagate Technology Llc Prevent data storage device circuitry swap
US9386267B1 (en) * 2012-02-14 2016-07-05 Arris Enterprises, Inc. Cooperative transcoding to multiple streams
US20160301959A1 (en) * 2013-11-13 2016-10-13 Lg Electronics Inc. Broadcast signal transmission method and apparatus for providing hdr broadcast service
US9661528B2 (en) 2004-12-23 2017-05-23 Electronic And Telecommunications Research Institute Apparatus for transmitting and receiving data to provide high-speed data communication and method thereof
US20170164033A1 (en) * 2014-08-07 2017-06-08 Sony Corporation Transmission device, transmission method, and reception device
US9679602B2 (en) 2006-06-14 2017-06-13 Seagate Technology Llc Disc drive circuitry swap
US10070141B2 (en) 2001-01-09 2018-09-04 Intel Corporation Method and apparatus for providing prediction mode scalability
US10123018B2 (en) 2015-09-29 2018-11-06 Dolby Laboratories Licensing Corporation Feature based bitrate allocation in non-backward compatible multi-layer codec via machine learning
US20190158895A1 (en) * 2016-03-21 2019-05-23 Lg Electronics Inc. Broadcast signal transmitting/receiving device and method
US10305536B2 (en) 1999-05-31 2019-05-28 Electronics And Telecommunications Research Institute Apparatus and method for modulating data message by employing orthogonal variable spreading factor (OVSF) codes in mobile communication system
US10863203B2 (en) 2007-04-18 2020-12-08 Dolby Laboratories Licensing Corporation Decoding multi-layer images
US10958915B2 (en) 2012-01-30 2021-03-23 Qualcomm Incorporated Method of coding video and storing video content
US10992983B2 (en) * 2017-08-30 2021-04-27 Sagemcom Broadband Sas Method for recovering a target file of an operating software and device for use thereof
US11032549B2 (en) * 2018-07-26 2021-06-08 Google Llc Spatial layer rate allocation
US11350114B2 (en) * 2013-04-08 2022-05-31 Arris Enterprises Llc Signaling for addition or removal of layers in video coding
US20230037494A1 (en) * 2021-08-06 2023-02-09 Lenovo (Beijing) Limited High-speed real-time data transmission method and apparatus, device, and storage medium
US11606528B2 (en) * 2018-01-03 2023-03-14 Saturn Licensing Llc Advanced television systems committee (ATSC) 3.0 latency-free display of content attribute
US11616995B2 (en) * 2020-05-25 2023-03-28 V-Nova International Limited Wireless data communication system and method

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7406176B2 (en) * 2003-04-01 2008-07-29 Microsoft Corporation Fully scalable encryption for scalable multimedia
FR2858741A1 (en) * 2003-08-07 2005-02-11 Thomson Licensing Sa DEVICE AND METHOD FOR COMPRESSING DIGITAL IMAGES
KR20060088461A (en) 2005-02-01 2006-08-04 엘지전자 주식회사 Method and apparatus for deriving motion vectors of macro blocks from motion vectors of pictures of base layer when encoding/decoding video signal
EP1869888B1 (en) 2005-04-13 2016-07-06 Nokia Technologies Oy Method, device and system for effectively coding and decoding of video data
WO2024000959A1 (en) * 2022-06-29 2024-01-04 华为技术有限公司 Data transmission method, electronic device, and medium

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0644695A2 (en) 1993-09-21 1995-03-22 AT&T Corp. Spatially scalable video encoding and decoding
EP0771119A2 (en) 1995-10-27 1997-05-02 Kabushiki Kaisha Toshiba Video encoding and decoding apparatus
GB2306846A (en) 1995-11-01 1997-05-07 Samsung Electronics Co Ltd Determining quantization interval in video signal encoder
EP0786902A1 (en) 1996-01-29 1997-07-30 Matsushita Electric Industrial Co., Ltd. Method and apparatus for changing resolution by direct DCT mapping
US5742892A (en) * 1995-04-18 1998-04-21 Sun Microsystems, Inc. Decoder for a software-implemented end-to-end scalable video delivery system
EP0883300A2 (en) 1997-06-05 1998-12-09 General Instrument Corporation Temporal and spatial scaleable coding for video object planes
US5852565A (en) * 1996-01-30 1998-12-22 Demografx Temporal and resolution layering in advanced television
WO1999012356A1 (en) 1997-08-29 1999-03-11 Siemens Aktiengesellschaft Method for compressing image information
WO1999033274A1 (en) 1997-12-19 1999-07-01 Kenneth Rose Scalable predictive coding method and apparatus

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6263022B1 (en) * 1999-07-06 2001-07-17 Philips Electronics North America Corp. System and method for fine granular scalable video with selective quality enhancement

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0644695A2 (en) 1993-09-21 1995-03-22 AT&T Corp. Spatially scalable video encoding and decoding
US5742892A (en) * 1995-04-18 1998-04-21 Sun Microsystems, Inc. Decoder for a software-implemented end-to-end scalable video delivery system
EP0771119A2 (en) 1995-10-27 1997-05-02 Kabushiki Kaisha Toshiba Video encoding and decoding apparatus
GB2306846A (en) 1995-11-01 1997-05-07 Samsung Electronics Co Ltd Determining quantization interval in video signal encoder
EP0786902A1 (en) 1996-01-29 1997-07-30 Matsushita Electric Industrial Co., Ltd. Method and apparatus for changing resolution by direct DCT mapping
US5852565A (en) * 1996-01-30 1998-12-22 Demografx Temporal and resolution layering in advanced television
EP0883300A2 (en) 1997-06-05 1998-12-09 General Instrument Corporation Temporal and spatial scaleable coding for video object planes
WO1999012356A1 (en) 1997-08-29 1999-03-11 Siemens Aktiengesellschaft Method for compressing image information
WO1999033274A1 (en) 1997-12-19 1999-07-01 Kenneth Rose Scalable predictive coding method and apparatus

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Bosveld F et al: "Hierarchical Coding of HDTV" Signal Processing Image Communication, NL, Elsevier Science Publishers, Amsterdam vol. 4, No. 3, Jun. 1, 1992, pp. 195-225.
NG S -B et al: "Two-Tier DPCM CODEC for Videoconferencing" IEEE Transactions on Communications, US, IEE Inc. New York vol. 37, No. 4, Apr. 1, 1999, pp. 380-386.
PHA 23, 726, U.S. Ser. No. 09/347,882, Filed: Jul. 6, 1999.

Cited By (119)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10305536B2 (en) 1999-05-31 2019-05-28 Electronics And Telecommunications Research Institute Apparatus and method for modulating data message by employing orthogonal variable spreading factor (OVSF) codes in mobile communication system
US6639943B1 (en) * 1999-11-23 2003-10-28 Koninklijke Philips Electronics N.V. Hybrid temporal-SNR fine granular scalability video coding
US7269785B1 (en) * 1999-12-30 2007-09-11 Genesis Microchip Inc. Digital manipulation of video in digital video player
US6798838B1 (en) * 2000-03-02 2004-09-28 Koninklijke Philips Electronics N.V. System and method for improving video transmission over a wireless network
US20020080878A1 (en) * 2000-10-12 2002-06-27 Webcast Technologies, Inc. Video apparatus and method for digital video enhancement
US7848433B1 (en) * 2000-11-22 2010-12-07 At&T Intellectual Property Ii, L.P. System and method for processing data with drift control
US20110103484A1 (en) * 2000-11-22 2011-05-05 At&T Intellectual Property Ii, L.P. Via Transfer From At&T Corp. Scalable Video Encoder/Decoder with Drift Control
US9578343B2 (en) 2000-11-22 2017-02-21 At&T Intellectual Property Ii, L.P. Scalable video encoder/decoder with drift control
US9313511B2 (en) 2000-11-22 2016-04-12 At&T Intellectual Property Ii, L.P. Scalable video encoder/decoder with drift control
US7974200B2 (en) 2000-11-29 2011-07-05 British Telecommunications Public Limited Company Transmitting and receiving real-time data
US10070141B2 (en) 2001-01-09 2018-09-04 Intel Corporation Method and apparatus for providing prediction mode scalability
US20020118743A1 (en) * 2001-02-28 2002-08-29 Hong Jiang Method, apparatus and system for multiple-layer scalable video coding
US20040261113A1 (en) * 2001-06-18 2004-12-23 Baldine-Brunel Paul Method of transmitting layered video-coded information
US8621532B2 (en) 2001-06-18 2013-12-31 At&T Intellectual Property Ii, L.P. Method of transmitting layered video-coded information
US20060200848A1 (en) * 2001-06-18 2006-09-07 At&T Corp. Method of transmitting layered video-coded information
US7958532B2 (en) * 2001-06-18 2011-06-07 At&T Intellectual Property Ii, L.P. Method of transmitting layered video-coded information
US6785334B2 (en) * 2001-08-15 2004-08-31 Koninklijke Philips Electronics N.V. Method for transmission control in hybrid temporal-SNR fine granular video coding
US20030035480A1 (en) * 2001-08-15 2003-02-20 Philips Electronics North America Corporation Method for transmission control in hybrid temporal-SNR fine granular video coding
US20050135476A1 (en) * 2002-01-30 2005-06-23 Philippe Gentric Streaming multimedia data over a network having a variable bandwith
US7483489B2 (en) * 2002-01-30 2009-01-27 Nxp B.V. Streaming multimedia data over a network having a variable bandwith
US20040071083A1 (en) * 2002-02-22 2004-04-15 Koninklijke Philips Electronics N.V. Method for streaming fine granular scalability coded video over an IP network
US20060133514A1 (en) * 2002-03-27 2006-06-22 Walker Matthew D Video coding and transmission
US8135852B2 (en) 2002-03-27 2012-03-13 British Telecommunications Public Limited Company Data streaming system and method
US8386631B2 (en) 2002-03-27 2013-02-26 British Telecommunications Plc Data streaming system and method
US7720999B2 (en) * 2002-11-26 2010-05-18 Qualcomm Incorporated System and method for optimizing multimedia compression using plural encoders
US20040103216A1 (en) * 2002-11-26 2004-05-27 Lane Richard D. System and method for optimizing multimedia compression using plural encoders
EP3203741A1 (en) * 2003-01-30 2017-08-09 Koninklijke Philips N.V. Video coding
US8005148B2 (en) 2003-01-30 2011-08-23 Koninklijke Philips Electronics N.V. Video coding
US9036715B2 (en) 2003-01-30 2015-05-19 Koninklijke Philips N.V. Video coding
US20060140269A1 (en) * 2003-01-30 2006-06-29 Bruls Wilhelmus Hendrikus A Video coding
WO2004068861A1 (en) * 2003-01-30 2004-08-12 Koninklijke Philips Electronics N.V. Video coding
US7761901B2 (en) 2003-03-19 2010-07-20 British Telecommunications Plc Data transmission
WO2004084520A1 (en) * 2003-03-19 2004-09-30 British Telecommunications Public Limited Company Data transmission over a network having initiallly undetermined transmission capacity
US9800885B2 (en) 2003-04-25 2017-10-24 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US11109048B2 (en) 2003-04-25 2021-08-31 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US9967580B2 (en) 2003-04-25 2018-05-08 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US9961355B2 (en) 2003-04-25 2018-05-01 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US9171577B1 (en) * 2003-04-25 2015-10-27 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US9854263B2 (en) 2003-04-25 2017-12-26 Gopro, Inc. Encoding and decoding selectively retrievable representations of video content
US20050169386A1 (en) * 2004-01-13 2005-08-04 Gerd Spalink Method for pre-processing digital data, digital to analog and analog to digital conversion system
US20050157794A1 (en) * 2004-01-16 2005-07-21 Samsung Electronics Co., Ltd. Scalable video encoding method and apparatus supporting closed-loop optimization
US20050180646A1 (en) * 2004-02-09 2005-08-18 Canon Kabushiki Kaisha Methods for sending and receiving an animation, and associated devices
US7426305B2 (en) 2004-02-09 2008-09-16 Canon Kabushiki Kaisha Methods for sending and receiving an animation, and associated devices
WO2006006835A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Scalable motion information encoding/decoding apparatus and method and scalable video encoding/decoding apparatus and method using them
US20060013309A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Video encoding and decoding methods and video encoder and decoder
US20060013306A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Motion information encoding/decoding apparatus and method and scalable video encoding/decoding apparatus and method employing them
WO2006006793A1 (en) * 2004-07-15 2006-01-19 Samsung Electronics Co., Ltd. Video encoding and decoding methods and video encoder and decoder
US9661528B2 (en) 2004-12-23 2017-05-23 Electronic And Telecommunications Research Institute Apparatus for transmitting and receiving data to provide high-speed data communication and method thereof
US20090316835A1 (en) * 2005-03-31 2009-12-24 Qualcomm Incorporated Power savings in hierarchically coded modulation
US8874998B2 (en) 2005-03-31 2014-10-28 Qualcomm Incorporated Power savings in hierarchically coded modulation
US8737470B2 (en) * 2005-03-31 2014-05-27 Qualcomm Incorporated Power savings in hierarchically coded modulation
US20100220816A1 (en) * 2005-03-31 2010-09-02 Qualcomm Incorporated Power savings in hierarchically coded modulation
WO2006108917A1 (en) * 2005-04-13 2006-10-19 Nokia Corporation Coding, storage and signalling of scalability information
US20060256851A1 (en) * 2005-04-13 2006-11-16 Nokia Corporation Coding, storage and signalling of scalability information
US20060233250A1 (en) * 2005-04-13 2006-10-19 Samsung Electronics Co., Ltd. Method and apparatus for encoding and decoding video signals in intra-base-layer prediction mode by selectively applying intra-coding
US8774266B2 (en) 2005-04-13 2014-07-08 Nokia Corporation Coding, storage and signalling of scalability information
US9332254B2 (en) 2005-04-13 2016-05-03 Nokia Technologies Oy Coding, storage and signalling of scalability information
WO2007011160A1 (en) * 2005-07-19 2007-01-25 Electronics And Telecommunications Research Institute Apparatus and method of embedded quantizaton for the improved snr scalbilty
US20080193033A1 (en) * 2005-07-19 2008-08-14 Hae Chul Choi Apparatus and Method of Embedded Quantization for the Improved Snr Scalbility
US8428380B2 (en) 2005-07-19 2013-04-23 Electronics And Telecommunications Research Institute Apparatus and method of embedded quantization for the improved SNR scalbility
US7734053B2 (en) * 2005-12-06 2010-06-08 Fujitsu Limited Encoding apparatus, encoding method, and computer product
US20070127585A1 (en) * 2005-12-06 2007-06-07 Fujitsu Limited Encoding apparatus, encoding method, and computer product
US9679602B2 (en) 2006-06-14 2017-06-13 Seagate Technology Llc Disc drive circuitry swap
US8630355B2 (en) * 2006-12-22 2014-01-14 Qualcomm Incorporated Multimedia data reorganization between base layer and enhancement layer
US20080152003A1 (en) * 2006-12-22 2008-06-26 Qualcomm Incorporated Multimedia data reorganization between base layer and enhancement layer
US11412265B2 (en) 2007-04-18 2022-08-09 Dolby Laboratories Licensing Corporaton Decoding multi-layer images
TWI488492B (en) * 2007-04-18 2015-06-11 Thomson Licensing Decoding apparatus
US10863203B2 (en) 2007-04-18 2020-12-08 Dolby Laboratories Licensing Corporation Decoding multi-layer images
US9305590B2 (en) 2007-10-16 2016-04-05 Seagate Technology Llc Prevent data storage device circuitry swap
US20090175358A1 (en) * 2008-01-03 2009-07-09 Broadcom Corporation Video processing system and transcoder for use with layered video coding and methods for use therewith
US8594191B2 (en) * 2008-01-03 2013-11-26 Broadcom Corporation Video processing system and transcoder for use with layered video coding and methods for use therewith
US20090279605A1 (en) * 2008-05-07 2009-11-12 Microsoft Corporation Encoding streaming media as a high bit rate layer, a low bit rate layer, and one or more intermediate bit rate layers
US8325800B2 (en) * 2008-05-07 2012-12-04 Microsoft Corporation Encoding streaming media as a high bit rate layer, a low bit rate layer, and one or more intermediate bit rate layers
US8379851B2 (en) 2008-05-12 2013-02-19 Microsoft Corporation Optimized client side rate control and indexed file layout for streaming media
US9571550B2 (en) 2008-05-12 2017-02-14 Microsoft Technology Licensing, Llc Optimized client side rate control and indexed file layout for streaming media
US8370887B2 (en) 2008-05-30 2013-02-05 Microsoft Corporation Media streaming with enhanced seek operation
US8819754B2 (en) 2008-05-30 2014-08-26 Microsoft Corporation Media streaming with enhanced seek operation
US8265140B2 (en) 2008-09-30 2012-09-11 Microsoft Corporation Fine-grained client-side control of scalable media delivery
US8380790B2 (en) 2008-12-15 2013-02-19 Microsoft Corporation Video conference rate matching
US20100153574A1 (en) * 2008-12-15 2010-06-17 Microsoft Corporation Video Conference Rate Matching
US20100149301A1 (en) * 2008-12-15 2010-06-17 Microsoft Corporation Video Conferencing Subscription Using Multiple Bit Rate Streams
US8947492B2 (en) * 2010-06-18 2015-02-03 Microsoft Corporation Combining multiple bit rate and scalable video coding
US20110310216A1 (en) * 2010-06-18 2011-12-22 Microsoft Corporation Combining multiple bit rate and scalable video coding
US8576271B2 (en) * 2010-06-25 2013-11-05 Microsoft Corporation Combining direct and routed communication in a video conference
US20110316965A1 (en) * 2010-06-25 2011-12-29 Microsoft Corporation Combining direct and routed communication in a video conference
US10284855B2 (en) 2011-01-18 2019-05-07 Dolby International Ab Video decoder with reduced dynamic range transform with inverse transform shifting memory
US20120183045A1 (en) * 2011-01-18 2012-07-19 Louis Joseph Kerofsky Video decoder with reduced dynamic range transform including clipping
US10652545B2 (en) 2011-01-18 2020-05-12 Dolby International Ab Video decoder with reduced dynamic range transform with inverse transform shifting memory
US9807395B2 (en) * 2011-01-18 2017-10-31 Dolby International Ab Video decoder with reduced dynamic range transform with inverse transform shifting memory
US10958910B2 (en) 2011-01-18 2021-03-23 Dolby International Ab Video decoder with reduced dynamic range transform with inverse transform shifting memory
US11431982B2 (en) 2011-01-18 2022-08-30 Dolby International Ab Video decoder with reduced dynamic range transform with inverse transform shifting memory
US20120183046A1 (en) * 2011-01-18 2012-07-19 Louis Joseph Kerofsky Video decoder with reduced dynamic range transform with inverse transform shifting memory
US9955165B2 (en) 2011-01-18 2018-04-24 Dolby International Ab Video decoder with reduced dynamic range transform with inverse transform shifting memory
US9848217B2 (en) * 2012-01-20 2017-12-19 Korea Electronics Technology Institute Method for transmitting and receiving program configuration information for scalable ultra high definition video service in hybrid transmission environment, and method and apparatus for effectively transmitting scalar layer information
US20150020131A1 (en) * 2012-01-20 2015-01-15 Korea Electronics Technology Institute Method for transmitting and receiving program configuration information for scalable ultra high definition video service in hybrid transmission environment, and method and apparatus for effectively transmitting scalar layer information
US20130195198A1 (en) * 2012-01-23 2013-08-01 Splashtop Inc. Remote protocol
US10958915B2 (en) 2012-01-30 2021-03-23 Qualcomm Incorporated Method of coding video and storing video content
US9386267B1 (en) * 2012-02-14 2016-07-05 Arris Enterprises, Inc. Cooperative transcoding to multiple streams
US20150036753A1 (en) * 2012-03-30 2015-02-05 Sony Corporation Image processing device and method, and recording medium
US11350114B2 (en) * 2013-04-08 2022-05-31 Arris Enterprises Llc Signaling for addition or removal of layers in video coding
US20220256176A1 (en) * 2013-04-08 2022-08-11 Arris Enterprises Llc Signaling for addition or removal of layers in video coding
US20150103887A1 (en) * 2013-10-14 2015-04-16 Qualcomm Incorporated Device and method for scalable coding of video information
US20160301959A1 (en) * 2013-11-13 2016-10-13 Lg Electronics Inc. Broadcast signal transmission method and apparatus for providing hdr broadcast service
US9736507B2 (en) * 2013-11-13 2017-08-15 Lg Electronics Inc. Broadcast signal transmission method and apparatus for providing HDR broadcast service
US20150172680A1 (en) * 2013-12-16 2015-06-18 Arris Enterprises, Inc. Producing an Output Need Parameter for an Encoder
US10397642B2 (en) * 2014-08-07 2019-08-27 Sony Corporation Transmission device, transmission method, and reception device
US20170164033A1 (en) * 2014-08-07 2017-06-08 Sony Corporation Transmission device, transmission method, and reception device
US10123018B2 (en) 2015-09-29 2018-11-06 Dolby Laboratories Licensing Corporation Feature based bitrate allocation in non-backward compatible multi-layer codec via machine learning
US20190158895A1 (en) * 2016-03-21 2019-05-23 Lg Electronics Inc. Broadcast signal transmitting/receiving device and method
US10750217B2 (en) * 2016-03-21 2020-08-18 Lg Electronics Inc. Broadcast signal transmitting/receiving device and method
US11178438B2 (en) * 2016-03-21 2021-11-16 Lg Electronics Inc. Broadcast signal transmitting/receiving device and method
US10992983B2 (en) * 2017-08-30 2021-04-27 Sagemcom Broadband Sas Method for recovering a target file of an operating software and device for use thereof
US11606528B2 (en) * 2018-01-03 2023-03-14 Saturn Licensing Llc Advanced television systems committee (ATSC) 3.0 latency-free display of content attribute
US11032549B2 (en) * 2018-07-26 2021-06-08 Google Llc Spatial layer rate allocation
US20210281850A1 (en) * 2018-07-26 2021-09-09 Google Llc Spatial Layer Rate Allocation
US11632555B2 (en) * 2018-07-26 2023-04-18 Google Llc Spatial layer rate allocation
US11616995B2 (en) * 2020-05-25 2023-03-28 V-Nova International Limited Wireless data communication system and method
US20230037494A1 (en) * 2021-08-06 2023-02-09 Lenovo (Beijing) Limited High-speed real-time data transmission method and apparatus, device, and storage medium
US11843812B2 (en) * 2021-08-06 2023-12-12 Lenovo (Beijing) Limited High-speed real-time data transmission method and apparatus, device, and storage medium

Also Published As

Publication number Publication date
JP2003533067A (en) 2003-11-05
EP1110405A1 (en) 2001-06-27
AU6560900A (en) 2001-01-22
CN1192629C (en) 2005-03-09
CN1321399A (en) 2001-11-07
WO2001003441A1 (en) 2001-01-11
KR20010086365A (en) 2001-09-10

Similar Documents

Publication Publication Date Title
US6501797B1 (en) System and method for improved fine granular scalable video using base layer coding information
US6480547B1 (en) System and method for encoding and decoding the residual signal for fine granular scalable video
US6788740B1 (en) System and method for encoding and decoding enhancement layer data using base layer quantization data
KR100676644B1 (en) System and method for scalable video coding
US6639943B1 (en) Hybrid temporal-SNR fine granular scalability video coding
Van Der Schaar et al. Adaptive motion-compensation fine-granular-scalability (AMC-FGS) for wireless video
US7016412B1 (en) System and method for dynamic adaptive decoding of scalable video to balance CPU load
US20040179606A1 (en) Method for transcoding fine-granular-scalability enhancement layer of video to minimized spatial variations
US8014451B2 (en) Video encoder/decoder with macroblock arrangement of significant item
CN101077011A (en) System and method for real-time transcoding of digital video for fine-granular scalability
KR100952185B1 (en) System and method for drift-free fractional multiple description channel coding of video using forward error correction codes
US20070121719A1 (en) System and method for combining advanced data partitioning and fine granularity scalability for efficient spatiotemporal-snr scalability video coding and streaming
KR20040036709A (en) Method for transmission control in hybrid temporal-snr fine granular video coding
KR20050061483A (en) Scalable video encoding
US6944346B2 (en) Efficiency FGST framework employing higher quality reference frames
Ishtiaq H. 263 scalable video coding and transmission at very low bitrates

Legal Events

Date Code Title Description
AS Assignment

Owner name: PHILIPS ELECTRONICS NORTH AMERICA CORPORATION, NEW

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:VAN DER SCHAAR, MIHAELA;CHEN, YINGWEI;RADHA, HAYDER;REEL/FRAME:010282/0004

Effective date: 19990817

AS Assignment

Owner name: KONINKLIJIKE PHILIPS ELECTRONIS N.V., NETHERLANDS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PHILIPS ELECTRONICS NORTH AMERICA CORPORTION;REEL/FRAME:013217/0455

Effective date: 20020805

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: IPG ELECTRONICS 503 LIMITED

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KONINKLIJKE PHILIPS ELECTRONICS N.V.;REEL/FRAME:022203/0791

Effective date: 20090130

Owner name: IPG ELECTRONICS 503 LIMITED, GUERNSEY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KONINKLIJKE PHILIPS ELECTRONICS N.V.;REEL/FRAME:022203/0791

Effective date: 20090130

FEPP Fee payment procedure

Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: FUNAI ELECTRIC CO., LTD., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:IPG ELECTRONICS 503 LIMITED;REEL/FRAME:027497/0001

Effective date: 20110824

FEPP Fee payment procedure

Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 12