The problem only happens in the evening and weekends. The verizon tech came around 2pm and tested it via the speedtest and it came back normal although he did note that the upload speed at 1-2 MB seemed slow. He basically said he didn't see any problems and there was nothing he could do and left.
He also said he had never heard of upload problems being an issue, only download speeds.