RBS will pay $1.1 billion to resolve claims that it sold toxic mortgage-backed securities to credit unions that later failed.