EDIT: All inputs & outputs are in stereo using 3.5mm TRS connections.

First some info about the project so that this all hopefully makes sense.

My setup: I have two audio outputs on my PC that I use simultaneously for different things, and two different devices to play those signals.

Example: Have a game on my headphones, while iTunes plays on my speakers.

Problem: Changing which output goes to which device is a hassle (something I do on a daily basis). I have to either change audio outputs in Windows, or physically change my cabling. It's also nice to sometimes have everything play from the same device.

Solution: A passive "summing mixer", with two inputs and two outputs, that allows me to push a button and change which device one of my PC's outputs go to.

The full schematic can be found here, but it basically goes like this:

INPUT JACK 1 -> VOLUME POT 1 -> RESISTORS -> SWITCH 1 -> OUTPUT JACK 1
INPUT JACK 2 -> VOLUME POT 2 -> RESISTORS -> SWITCH 2 -> OUTPUT JACK 2

If a switch is toggled, it directs that audio to the other output jack. This results in both inputs on one output if one is toggled, or a complete swap if both are.

My dilemma lies in not knowing how best to handle the multiple grounds. Currently, INPUT JACK 1, VOLUME POT 1, and OUTPUT JACK 1 all share ground, as do INPUT JACK 2, VOLUME POT 2, and OUTPUT JACK 2. One of my co-workers, who graduated as a sound engineer, recommended that I tie ALL of the grounds together and incorporate a ground lift switch on the inputs. I don't doubt his expertise, but his job doesn't involve PC or consumer level audio (it's entirely dealing with professional audio gear), so I'm cautious due to differences between consumer and pro audio equipment.

Did he get it right, or is there a better approach for what I'm doing?
If he is correct, should I tie my PCB's ground plane into the common ground?