IFS says EU single market membership is worth an additional 4 percent to U.K. economy.