A software program dubbed "Mayhem" was poised to win the final round of a three-year contest to teach computers to launch and defend against cyber attacks.