You don’t have to wear white. That’s a myth. Only the players must wear white. You can show up in a neon green tracksuit and a top hat. Please don’t. But you can.
However, there is a de facto rule: wear something smart-casual. You’ll see blazers, sundresses, and boat shoes. You’ll also see a guy in a giant novelty hat shaped like a tennis ball. He’s allowed. Just don’t block anyone’s view.
If you’re in the Royal Box? Then yes—strict dress code. Men must wear a jacket and tie. Women must wear a hat. No trouser suits if you’re a woman? Actually, that rule was quietly dropped. Progress!