XSplit have released a better x264 encoder in their update package. The directions in this thread are no longer necessary, simply update to the latest XSplit version. More information: http://www.xsplit.com/blog.php?post_id=267
The content below is preserved for historical reasons only.
I recently wrote a stream quality guide and lambasted XSplit for its lack of multi-core encoding which severely hinders performance on modern PCs that can execute 4 or 8 threads simultaneously. I was intrigued since XSplit uses x264, which is programmed to be multi-thread capable and decided to look into it further.
Turns out XSplit ships with an old x264 encoder which either doesn't support multithreading or it is turned off by default. Thankfully due to the wonders of API compatibility, we can simply drop in a replacement encoder and get full benefit from multi-core systems! Here are two videos highlighting the difference, encoded with the exact same xsplit settings - same bitrate, same FPS, same quality - only the modified encoder.
Here's a preview of the difference using fast action (Quake II) with hard-to-encode scenes (motion blur, quick camera switching, etc):
Old x264: http://www.mediafire.com/?vivb4c3dckabuy3
New x264: http://www.mediafire.com/?4wz96f7i84sk8cx
How To Install
0. Close XSplit.
1. Download libx264112.zip.
2. Rename libx264-112.dll in your C:\Program Files\SplitMediaLabs\XSplit folder to libx264-112.dll.old
3. Extract the new libx264-112.dll to your C:\Program Files\SplitMediaLabs\XSplit
That's it! Now when you encode with XSplit, it will use as many cores as necessary for a good quality video.
On my Core i5 2500k @ 4GHz, I am able to use the x264 "medium" preset - the same quality that our TSL VODs are encoded in - in realtime @ 720p30. Obviously results will vary once you throw SC2 into the mix, but for those of you who use XSplit on quad core and especially i7 CPUs, this should allow you to greatly increase quality on your stream (note that replacing this DLL won't actually improve quality by itself, it will allow you to set higher quality without running into single thread CPU limits).
The x264 encoder was taken from LoRd_MuldeR's libx264 for AVIDemux replacements (link). I have re-hosted it on TL to ensure a safe download.