A Brexit would cause building costs to spike, exacerbate the industry's skills shortage and slow down the delivery of affordable housing, CEO warns.