Thursday, February 19, 2009

Fixing PS3 WIFI video pausing and other glitches.

I recently ran into an issue streaming video to my PS3. I had TCP re-transmits ( killing the UPNP connection ) and video pausing occuring destroying any enjoyment of my streamed video. I also use XMBC as a client and I can say that in most cases it's the PS3 firmware it's not buffering enough or is just too sensitive to lag, given the same server will work better with XBMC. The other thing is that you really have to transcode just to rate limit the data stream to around 4-6mb for any consistancy in playback. If you don't transcode, hit the triangle key and check the bitrates out, you'll see it burst upwards of 25mb or more and much choppier playback and long delays.

My gut feeling is that the latest PS3 firmware as of March 2009 is the socket buffering and app level video buffering or a combination is causing the problem. It's also sensitive to drops of UPNP server with cryptic errors messages ( really annoying ). If it is a result of our server setup then there are a few things we can do ...

So first a word on my setup, I have a Via C7 server running mediatomb ( UPNP server ) transmitting via a router to my PS3 over 802.1g. I also run the Java based PS3 MediaServer which is a zero config service highly recommended. The simple issue is that transcoding must be enabled to ensure consistant playback on the PS3 but I'll go through all optimization tricks first ...

My network is something like this ...

[Server] <.... 100 BaseT Ethernet wired ...> [ WiFi Router ] <..... 802.1g ...> [PS3]

Potential problems with over the air video transmission

1. Wireless packets lost due to channel interference, noise etc.

- 5 or so neighbour WiFi routers , nothing too serious
- I choose a channel far away from '6'!
- I boost power output to 80 milliwatts
- I have a mixed mode Wifi net. b/g/n devices are active
- microwave ovens, wireless phones,wireless keyboard, bluetooth etc.

So I have a few electronic gizmo's that could all interfere with the signal. I'm kind of stuck with what I have from a wireless perspective and still notice video pausing. tcpdump captures confirm TCP resets, there is no 802.11g packet drops though so it's down to buffering, over the air corruption causing my issue. So 54mbps get's me 6.75MB per second thruput in theory, reality is probably less than half this around 3-4MB per second. If you press the 'triangle' button when streaming a video on the PS3 the top bar will show the stream rate. It's handy because it will give you a rough indication of the bursty data that is likely causing delays/pauses.

I saw lots of pauses/drops with TVersity none transcoded camera video ( m-jpeg ) ~35mbps rates.

I saw less pauses but still a few with divx encoded files that burst from 4-16mbps

I saw practically none when transcoding and enforcing bit rates to keep it below 6mbps.
audio rate= 192k
bit rate=4096k

Regardless of the root cause of these issues, there is a pattern the higher the bit rate, the increase in pausing and connection drops occur. The issues disappear when I enforce the transcoding. I tested streams on the laptop running XMBC this allows you to eliminate problems with PS3 and server ( it was also pausing ... ) but in general I found XBMC slightly better as a client.

There is also likely some performance issues with my setup so first lets get rid of the obvious issues like file access times, transmit delays e.g. buffer sizes ...

Kernel Tweaking using sysctl.

To modify the scheduling we'll have to rebuild the kernel, we'll do this later but it's not the bottleneck or problem for my wireless packet issues. We can use a server edition of Linux or compile the kernel for server mode. First lets do some hacking of the TCP stack and file system just to try and get some performance boost at the cost of time stamping, memory consumption etc.

TCP read/write buffering is 4k by default in Linux and the max TCP out of order counts is 3. So lets beef up my TCP buffers and allow a few more out of order conditions to persist before a TCP reset occurs ... Do a Google search on this to get a rough idea of what is being modified, then come back and randomly hack away at your setup just for fun ... test video's and other services to ensure you haven't bust anything... So you have some kernel parameters and values to changes lets head to the file /etc/sysctl.conf and change the read/write buffers for TCP

net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ip4v.tcp_reordering = 6

I also want to reduce the overall level of 'swapping' to disk. swappiness default is 60 in Linux 2.6. To truly test this, you'd have to run some heaving execution over time. Also don't modify it if you have less than 256mb probably pointless ...

vm.swappiness = 40

Not required but you can change the default TCP algorithm from 'reno' to others that offer superior TCP stack behaviour. It's not an issue I don't think unless your handling thousands of TCP connections and want to maximize performance but you can change the TCP algorithm:

net.ip4v.tcp_congestion_control = cubic

Filesystem tweaking

I also could do with tuning up my filesystem so access times are faster, rather than the default mount settings. The biggest change that everybody insists you should do , is to remove the access time stamping on the filesystem. This is easy, you basically add the option 'noatime' to your mount and voila you should see a big boost in performance of the disk I/O. So in your /etc/fstab file you can pick your mount you'd like to tweak e.g.

$ umount /data
/dev/sda2 /data ext3 noatime,nouser,exec,suid,errors=remount-ro 0 2
$ mount /data

Optimize directory structure, indexing ...

man e2fsck and look at the -d and -f options for details. These options will order your disk directory indexes more efficiently and it's a quick and dirty method of getting some bang for buck performance ...
$ umount /data
e2fsck -d -f /dev/sda2
$ mount /data

Recompile your kernel and optimize for server mode operations

This will get you a few boost points. First we need to check what board/cpu and card configuration we have so we can enable all the associated drivers by default ( faster boot/initialization ). We then turn on some options that should make it run a little faster than a standard distro for server use at least. I'm still using the AS scheduler but there are some 'server' options you can set, enable all board drivers etc. The only thing I noticed after kernel recompile was server data transfer rates increased significantly, torrentflux downloads where pretty fast with barely any CPU utilization, VLC reading over Samba I was getting 7.5mb fairly constant read rates and demuxing at 2.5mb didn't try HD but you get the idea. There's lots of articles on how to do this:

Kernel Compiling: What' devices are sitting on my bus, what processor do I have ?

# lspci
# cat /proc/cpuinfo

You've probably got SATA drives attached. They'll be mounted as /dev/sda ... /dev/sdb etc. To tune the access performance of this we can use 'sdparm'. Don't use 'hdparm' you'll likely see IOCTL errors ( errors accessing the kernel resources ). Your SATA drive should probably be DMA mode 6 or some other high value. Again do some reading but in short sdparm is likely the tool you want if it's a new system... if you have old IDE drives then hdparm is your tool.

Kernel Compile: know your board,cpu,drive options and intended use of setup...
I have a Via C7 Esther processor VIA supporting chipset. So I'd like to enable 'y' to the VIA associated flags. I probably could make the SATA,PATA support embedded but it's fine as a module for me ... cat /boot/config will show the current kernel compile options. For me I see ...

plus a bunch more ...

My kernel will compile optimized for the C7 chipset. Other options we'll want is the scheduler tuned not for pre-emption but for dumb processing, we want threads to hog the CPU since we'd like our video transcoding, UPNP serving to be atomic and not sliced up for other processes... we need greedy thread execution.

Post Options