In:
Section "ServerLayout"
You have:
InputDevice "USB Mouse" "CorePointer"
So ensure that's in the XF86Config file.

Then make sure you have:
Section "InputDevice"
Identifier "USB Mouse"
Driver "mouse"
Option "Device" "/dev/input/mice"
Option "SendCoreEvents" "true"
Option "Protocol" "IMPS/2"
Option "ZAxisMapping" "4 5"
Option "Buttons" "5"
EndSection

Which is the section that is referenced by the Server Layout section.

Give that a go and see if it works.