A comparative study of adaptive degradation of video quality over the internet

1. A comparative study of adaptive degradation of video quality over the internet Shafquat Mallik

2. Introduction • Establishing need for graceful degradation • Video Streaming • Requirements and nature of internet • Various Techniques • Error Control Mechanism (None requires QoS aware network) • Synchronous: • Uses time. • Checksum appended • Receiver sends an ack. • Asynchronous • Uses Space • FEC • No feedback required • Congestion Control • Receiver Based • Data divided into layers • Receiver decides how many layers it can join. • QoS aware nw can drop low priority layers • Sender Based • Feedback mechanism is used from all receivers

3. Initial Work • PET • ULP • FEC • Layering

4. Perceived Quality and bandwidth characterization of layered MPEG-2 video encoding. • Various layering mechanisms are considered to show how layers should be created and prioritized to maximize the perceived quality over a given range of network conditions.

5. Various Layering Schemes • Data Partitioning • Divides the encoded bit stream into two or more layers by allocating the data in bit stream to various layers. • PBP are introduced to specify the grouping of DCT coefficients into the layers. • Easiest to implement. • SNR Scalability • Encodes the video using a quantizer scale Qb to generate the base layer and then using the difference between original video and base layer using a quantizer scale Qe to generate the enhancement layer.

6. Approach of the paper • All possible layering structures were considered and out of them the structure that offered best quality bandwidth trade off was selected.

7. Results

9. 2 Layer DP with random packet loss in enhancement layer

10. Packet Loss Resilient Internet Video Streaming[1] • This scheme uses a scalable video coder combined with unequal error protection applied across packets. • Scalable coder produces a bit stream, decodable at different bit rates. • Markov model with two states is used to approximate the bursty packet loss behavior • State G (Good) packets are received correctly and timely • State B (Bad) Packets are lost either due to congestion or delayed arrival. • Average Loss probability PB = Pr(B) = PGB /(PGB+ PBG) • Average Burst Length LB = 1/PBG

11. FEC • FEC scheme uses Reed-Solomon code across packets. • At the sender copies of sent information is stored until k packet have been sent. • Then n-k redundant packets are calculated and transmitted. • These n packets are called block of packets (BOP). • To maintain a constant data rate on the channel the source data rate must be reduced by the factor k/n • At receiver all information can be reconstructed from any subset of at least k correctly received packets • Decoding process can start as soon as any k packets of a BOP are received. • Residual Block Error: • Probability that a BOP cannot be reconstructed. i.e. more than (n-k) packets are lost within a BOP. • It is summation of probability of m lost packets within a block of n packets P(m,n). • P(m,n) determines the performance of FEC scheme.

12. Scalable Video Coding with unequal error protection • Given average packet loss rate and average burst length a video transmission scheme can be designed to guarantee a certain BER. • Two layer scalable video coder is used that uses H.263 base layer encoding and an enhancement layer. • Both layers are packetized into one BOP along with redundancy. • For error concealment if less than k packets are received, for enhancement layer missing image parts are replaced by spatially interpolated image content from the base layer. • If packet loss occurs in base layer, enhancement layer is switched off.

13. PSNR Estimation • PSNR is used for quality estimation. • Probability that layer l is displayed is • Probability that base layer is also affected by packet loss is PL . let PSNRl be the average PSNR that layer l is displayed over the whole time period encoded in one BOP. Then the average PSNR resulting from switch off strategy is

14. Simulation Results • Input sequence are foreman and silent voice encoded at 30 fps. • 15% redundancy was allowed. • Proposed scheme achieves graceful degradation.

15. Unequal Loss protection: Graceful degradation of Image Quality over Packet Erasure Channels Through Forward Error Correction[2] • Unequal amount of FEC is applied to progressive data to provide graceful degradation in case of packet loss.

16. Formalizing the Framework • Message M (a sequence of data) • mi number of data bytes assigned to stream I • fi = N – mi number of FEC bytes assigned to stream i. • Redundancy assignment is an L dimension FEC vector whose entries are length of FEC assigned to each stream. • ƒ = (ƒ1, ƒ2,…. ƒL) • For a given ƒ M is divided into Mi(ƒ) fragments and define it to be sequence of data bytes in the ith stream. • Prefix of M containing the first j fragments for redundancy vector ƒ as • M(j, ƒ) = M1(ƒ) M1(ƒ) …Mj(ƒ) • Incremental PSNR of stream i is • gi = PSNR [ M (i, ƒ) ] – PSNR [ M (i-1, ƒ)] • The quantity c(ƒi) is the probability that receiver can decode stream I • Expected PSNR would be • G(ƒ) = Σ c(ƒi) gi(ƒ) • A value of ƒ that maximize G(ƒ) subject to packet loss estimate Pn is required.

17. Algorithm For Solving the ULP problem • Since the globally optimal solution is computationally prohibitive for a useful amount of data, a local search hill algorithm is used that makes assumptions about data but is computationally tractable. • Initialize each stream with data bytes only such that mi = N and ƒi =0,1,2,3…L • In each iteration algorithm examines a number of possible assignments equal to 2QL, where Q is the maximum number of FEC bytes that can be added or subtracted to a stream in one iteration. And L is the number of streams. • G(ƒ) is determined after adding or subtracting 1 to Q bytes of FEC data to each stream. ƒ corresponding to the highest G(ƒ) is chosen. • Repeat the search until no more value improves the expected PSNR.

18. Results • Standard 512*512 and 256*256 gray scale image compressed with SPHIT is used. • Channel loss model is exponential PMF with a mean loss rate of 20%. • ULP and ELP were compared. • ULP and ELP gave an expected PSNR of 29.42 and 28.94 respectively. • Under good channel conditions ULP performs better than ELP. • ULP degrades gracefully whereas ELP has a sharp transition at loss rate near 51% • ULP out performs ELP 85% of the time. • System provides progressive transmission once a number of packets equal to number of data bytes in packet stream 1 are received decoding of image can begin and as when additional packets are arrived image can improve.

20. Efficient partitioning of unequal error protected MPEG video streams for multiple channel transmissions • Efficiently partitioning forward error protected, pre-encoded video data for transmission over multiple channels. • An algorithm is proposed which efficiently partitions FEC-protected video data across multiple channels to • minimize the bandwidth consumption and when the aggregate bandwidth is sufficient to transmit the video or • maximize the decodable frame rate when aggregate bandwidth is limited.

21. Transmission Model • Multiple channels are used to provide a logical channel with sufficient or nearly sufficient bandwidth. • Assume there are N channels and each is parameterized by BW, Delay (T), Packet loss rate (PLR) • FEC across packets is applied to each coded frame to recover lost frames. • Each channel is protected at a different level because of different PLR • Assume the channel conditions vary in order of TG seconds and interval of channel estimates is of multiple GOP durations.

22. Optimal Partitioning Problem • A GOP consisting of Q frames is to be transmitted over N channels with different bandwidths, PLRs, and latencies. • Let • S is the set of all the frames in a GOP. • Si(n) set of frames allocated to channel i. • Si(n)∩ Sj(n) = Φ if i <> j • S1(n) U S2(n)…. SN(n) = S • F(Aj) is the non-FEC coded data size of frame Aj of type A. Redundancy is added in the fractional amount of factor Ri(A,n) • Total amount of data to be allocated to a channel is • Ei = ∑ ( 1 + Ri ( A,n ) ) . F ( A j )

23. Achieving Minimum Bandwidth • Assume channels are sorted in ascending order of reliability. • Assume BW of channel 1 is unconstrained • More reliable channels 2 thru N are unable to transmit the entire FEC coded GOP within TG seconds. • Goal is to minimize the required BW of channel 1 with the constraints of channel 2 to N bandwidths. • Let the solution be partition S = Ś1UŚ2U…..ŚN that gives min E1 subject to Ei ≤ BWi,. TG

24. Achieving Maximum Frame Rate • Objective is to transmit as many frames as possible in one GOP within limited bandwidth-time product. • Frames in GOP are reordered based on their encoding dependencies. • Reordering pattern can be determined prior to the start of transmission for fixed GOP patterns. • Optimizing problem is to maximize the number q of transmitted frames. such that the Ei ≤ BWi,. TG

25. Pruned Tree Search Algorithm • Search space is O(NQ) for the solution to the optimization problems. • Tree depth corresponds to number of frames and branches represent channel assignments. Constraints are associated with each node. • Impossible situations are pruned • If channel estimates are inaccurate greedily either frames are dropped or added at the end.

26. Experiments and Results • Minimizing BW consumption: • Experiments show that formulation achieves a high BW utilization of channels 2 to 3 and PTS finds the optimal solutions as found by a full search. • The algorithm is slowest when equal bandwidth is assigned to channels because pruned tree has more balanced branches. • Computation time reduces by a factor of 5-10 when channel BW are very unequal.

27. Maximizing Frame Rate • Comparison with the Greedy Partitioning: • With total available bandwidth ranging from 70% to 150% of video average bit rate and three channels of equal bandwidth. • Proposed system outperforms greedy allocation with maximum difference from 0.64 to 2.12 frames/GOP and minimum range from -0.07 to 0.64 frames/GOP. • Effect of Differing Video Content: • Various video content generates different patterns of frame size traces. • Effect of Channel Parameter Mismatch: • Performance of the transmission model depends on the accuracy of the channel parameters provided. • At 130% estimated normalized bandwidth the average frame rate drops by 6-7 frames/GOP when actual bandwidth is -15% lower than the estimates and increase by 1-2 frames/GOP when the actual value is +5% higher.

28. An adaptive Protection Framework for MPEG Base Layer Video Streaming Over Internet • After reinvestigating the PET an adaptive optimization based protection assignment framework for streaming MPEG base layer under a bandwidth constraint.

29. An assignment is a vector of X of dimension m, • X= (x1, x2, … xm), for all i 1≤xi≤N for message fragment I • Algorithm calculates the value xi which determines the amount of protection associated with fragment • Amount of protection used for fragment is ri*N/xi • For x1= r1/L all parity data is assigned to protect the first frame only. • General problem is to find an assignment that optimizes expected received quality under a given rate budget R, satisfying the constraint x1≤x2≤x3≤….xm

30. PET ULP system • Each message is an independent video subsequence representing the boundary of error propagation. • N is the number of packets used to carry the encoded GOP data and FEC parity data. • L is the size of each individual packet. • A message is partitioned into m partitions each of size ri • A priority is assigned to each message and message is protected according to its priority. • Given algorithm calculates xi which is amount of protection associated with each fragment i. • Amount of protection used for fragment is (ri*N/xi) • Problem is to find an assignment that optimizes expected received quality under the given rate budget R, satisfying the constraint • x2≤ x3≤ x4≤ x5≤….. xm

31. Definition of Utility-cost Function • Defined for each independent GOP. • Cost for achieving quality at a given level i is the cumulative bit-rate required to correctly decode the message from the lowest level to the ith level. • Utility at that level is the quality gained from correctly decoding the message segment up to that level. • Quality is defined as average PSNR value computed over the entire GOP. • Utility cost function is computed in two steps. • Designate number of levels and the frame loss compensation pattern at each level. • A utility is computed according to the designated frame_loss_compensation pattern for each level in terms of PSNR considering • Order of frame importance. (I>P>B) • Dependency between frames. • Importance of data. (Motion vector> Texture data…)

32. Adaptive Optimization Assignment Framework • The system is designed to operate in real time in response to feedback of conditions of the transmission channels. • Inputs to module are N number of packets to be used by ULP system, channel error probability distribution, bandwidth budget for a particular GOP, Utility cost function for the GOP

33. The optimization algorithm searches for the assignment that maximizes E[U] • When number of levels is small an exhaustive search is sufficient. • For large number of levels a pruning algorithm is used that prunes off infeasible regions, then exhaustively searches for best assignment in feasible region.

34. Experiments & Results • Experiment 1: • Comparison of PET with optimized PET • At six different mean packet loss rate scenarios: 0.05, 0.1, 0.15, 0.2, 0.25, and 0.3. • Optimized PET outperforms original by over 1 DB.

35. Experiment 2: • Varying bandwidth to evaluate adaptation. • Redundancy available is 10%, 20%, 30%, 40%, 50% and 60% over original data. • Original PET has over 30% • Optimized system outperforms original.

36. H.263 Standard[5] • Low bit rate DCT based, Motion compensated coding on internet and wireless networks. • Frame structure: • Supported formats QCIF, sub-QCIF, CIF, 4CIF, 16CIF. Frame is split into 16*16 pixel MB. • Motion Compensation: • Inter (I) frame containing only intra macro blocks. First motion compensated and then the difference is encoded • Intra (P) frame containing both intra and inter macro blocks. Coded independently. • Transform and Entropy Coding • 8*8 block is transformed using DCT. • This block is scanned into a one dimensional array ordered from low to high frequency. • Run length coding is applied.

37. Unequal Loss Protection (ULP) • ULP framework operates by applying unequal amount of FEC to data that are compressed with an unmodified embedded algorithm and transmitted over lossy packet networks. • Based on priority encoding transmission that assigns unequal amount of FEC to data sent over lossy packet networks. • The packet is divided into L streams and each contains symbols from each of N packets. Given stream can be decoded successfully as long as number of lost packets is less than or equal to number of FEC symbols.

38. ULP algorithm operates in the form of a utility curve function. Utility is how much benefit the user is likely to enjoy by receiving and decoding the stream and Cost is number of bits used to achieve a given utility.

39. Applying ULP to H.263 • Several changes to original ULP system are required. • Create Independent Subsequences. • GOP size is influenced by application of interest. • Reorder the compressed bit stream. • First I frame is a GOP • Picture headers from all frames a GOP • MB headers from all frames in a GOP…. • Select the encoding bit rate. • Generate a family of utility-cost curves.

40. Results & Methodology • Expected frame PSNR. • Results show that the expected performance of ULP system is much better than baseline H.263 due to packet loss protection used by ULP. • Frame PSNR at various packet loss rates: • No packet loss H.263 performs better since the encoding is at the target bit rate. Whereas ULP uses a lower bit rate with redundancy. • At mean loss rate (9.38%) ULP provides a better quality where H.263 has an unusable video sequence. • At very high loss rate (40.63) ULP still provides a good quality which is only slightly degraded.

42. Multicasting of Adaptively-encoded MPEG4 over QoS-aware IP networks. Multicast congestion control for single rate (sender based) single source IP multicast sessions over a QoS aware IP network.

43. Multicast congestion control techniques • Single rate: (Sender based) • Single rate is sent to all receivers. • Scalable Feedback Control uses messages from receivers to estimate group reception • Scalability • Choosing group representative. • Single slow receiver can drag down the data rate for whole group. • Multi rate: (Receiver Based) • Generate the source data in layered format. Receiver decides how many layers it can join. • Best effort network can drop packets in the base layer making enhancement layers useless.

44. RED (Random Early Detection) • A buffer management technique used for congestion avoidance in IP networks. • Detect upcoming congestion by computing an average of queue size in the router. • IF queue-average < min • THEN no action taken • IF queue-average > min AND queue-average < max • THEN with probability drop packet • IF queue-average > max • THEN drop packet.

45. Backward Explicit Congestion Notification • A scalable mechanism provided by the routers than receivers. • No overhead of selecting representative. • Compatible with a variety of transport layer protocols. • If Queue size is between min and max a packet is marked and sent to receiver, and receiver marks a flag in ACK message and send it to sender • Sender reduces the congestion window and slow start threshold. And sends a notification to receiver. • Coupled with TCP • Takes a round trip time before sender reacts. • BECN send the notification even when queue is > max.

46. MPEG 4 is used because it can encode real time video adaptively to target a certain rate. • Encoded video can be generated in layers. Network Model • Network should support priority dropping of packets in time of congestion • Provide congestion notification from router to the multicast sender.

47. Flow control Algorithm • REPEAT every t seconds • IF NoFeedback message received • THEN • increase Total_Rate • Reduce basicLayerPortion • Reduce t • ELSEIF Feedback increasing at all priority levels • THEN • reduce TotalRate • Reduce t • ELSEIF Feedback increasing at all Basic layer • THEN • Reduce basicLayerPortion • Reduce t • END REPEAT

48. END to END Architecture • Sender marks the packet with two different priorities, with basic information marked with higher priority and enhancement info with lower priority. • While congestion is developing routers that run RED will send back BECN messages to sender with information on priority level that causes congestion. • Based on the rate of these feedback messages the sender runs an algorithm to search for an operating point that will reduce this feedback message rate. • Sender will try to match the sending rate for the high priority with the sending rate of slow recievers.

49. Simulation and Performance

50. Results