Was caused by using Apple Airport Extreme and DHCP/NAT. Never got it to work instead put my old Dovado firewall "before" Airport Extremes and set them to bridged instead.
So no more laggy VPN tunnels

EDIT: tried to get the AEs to work but Apple is well not a very configurable product