The Search for Unintended Consequences

Any idiot can build a system. Any amateur can make it perform. Professionals think about how a system will fail.  It’s very common for people to think about how a system will work if it is used the way they imagine. But they don’t think about how that system might work if it were used by a bad actor or a perfectly ordinary person who is just a little different from what the person designing it is like.

Companies need to be thinking about how each product could actually be used in the real world. If you build a product that works great for men and is going to lead to harassment of women, you have a problem. If you build a product that makes everyone’s address books 5 percent more efficient and then gets three people killed because it their personal information to their stalkers, that’s a problem.

What you need is a very diverse working group that can recognize a wide range of problems, that knows which questions to ask and has support inside the company and in the broader community to surface these issues and make sure they are taken seriously. If they’re in there from day one it makes a huge difference.

Former Google engineer Yonatan Zunger in an interview with NPR