The U.K. government is being told to increase spending on infrastructure projects to help offset any Brexit related slowdown.