The number of British air passengers could drop by 3 to 5 percent by 2020, the International Air Transport Association said.