So I have a TTL to USB adapter this time instead of a TTL to Serial, then Serial to USB adapter and it isn't acting as fast as I think it should.  The chip is a FTDI chip and I did see an interesting performance boost messing with an option.

This was at 16 by default and it caused everything to run at about 700 milliseconds where before I was at 10 milliseconds per cycle.  By chancing it to 1 millisecond, I'm at about 15'ish ms per cycle, not perfect but not horrible.

I hope that is something I can set programmatically.  Not sure if any of the other options may be directly effecting speed too.