After I did a couple different permutations, I found that I didn't have the fuses properly programmed.
By using 8 Mhz with a 4ms delay, I got the board to actually work. I have yet to try it with a 0ms delay yet.
Time for a quote on the remaining LEDs I need I do believe