Back to writing

Gradient Descent for Life

Gradient descent is a method used in machine learning to find better answers.

You begin somewhere on a landscape. You measure the slope beneath you, take a step downhill, measure again, and repeat. Eventually, you arrive at the bottom of a valley—or at least somewhere that appears to be the bottom.

This is useful when training an AI model. It may also be a useful way to think about learning, careers, and opportunity.

The obvious lesson is to keep taking small steps toward improvement. But that interpretation misses the most interesting part.

The real question is: Which valley are you descending into?

The problem with local minima

Imagine being dropped onto a vast mountain range in thick fog. You can see the ground immediately around you, but not the wider landscape.

If you always walk downhill, you will eventually reach a low point. Yet it may only be the bottom of a shallow valley. A much deeper and more interesting valley could exist just beyond the next ridge.

In optimization, this is the problem of a local minimum: a solution that looks optimal from your current position but is not the best solution available across the entire landscape.

Life contains local minima too.

You become competent at a skill, receive some recognition, and find increasingly predictable ways to earn rewards from it. The slope points clearly downward, so you keep following it. Each step improves your position within that particular valley.

Eventually, you may become extremely well optimized for an opportunity you selected before you understood the landscape.

Explore before you descend

A purely efficient learner would choose one subject and study it continuously. They would avoid distraction, ignore adjacent fields, and accumulate depth as quickly as possible.

That works if they have already chosen the right valley.

Early in the learning process, however, they rarely know enough to make that choice. The landscape is still hidden. What looks like distraction may actually be reconnaissance.

This suggests a different strategy: go deep on many things early enough to discover what each landscape contains.

“Deep” matters here. Sampling a subject superficially often reveals only its vocabulary and stereotypes. You need to move far enough into a field to encounter its underlying structures, difficult questions, and unusual people.

You do not need mastery. You need enough depth to feel the slope.

Learn to program well enough to build something real. Study economics until incentives begin appearing everywhere. Learn design until you notice hierarchy and friction. Run a business until abstract ideas about markets collide with payroll, customers, and limited time.

Each excursion gives you a new starting point and reveals another possible valley.

Depth creates optionality

Broad exploration is often confused with indecision. But purposeful exploration does not mean drifting endlessly from one novelty to another.

Its purpose is to build a map.

Each field you explore gives you:

  • New mental models
  • Better questions
  • Exposure to different kinds of problems
  • Relationships with people who see the world differently
  • Evidence about what sustains your curiosity
  • Clues about where your abilities may compound

The value is not only in choosing among these fields. It also lies in combining them.

Some of the richest valleys sit between established disciplines. Marketing becomes more interesting when combined with software. Software changes when combined with organizational design. AI becomes more useful when grounded in the messy realities of operating a company.

A person with depth in several areas can see opportunities invisible to someone whose knowledge follows a single narrow path.

Your learning rate matters

In gradient descent, the learning rate determines the size of each step.

If it is too small, progress is reliable but painfully slow. If it is too large, the process can overshoot promising solutions or become unstable.

People have learning rates too.

Small steps are appropriate when mistakes are expensive or when you are refining a skill you already understand. But early exploration often benefits from larger moves: building a project, changing environments, publishing an argument, starting a company, or attempting work before you feel fully qualified.

These moves generate stronger feedback than passive study.

Reading ten books about entrepreneurship may move you gently downhill. Trying to sell something shows you the shape of the terrain.

The right learning rate changes over time. Exploration calls for larger, faster experiments. Exploitation calls for smaller, more deliberate refinements.

Add noise on purpose

Optimization systems sometimes use randomness to avoid getting trapped. Noise can knock the system out of a comfortable position and expose it to another part of the landscape.

In life, we often treat randomness as interference. We optimize our feeds, routines, professional networks, and information sources until they reliably reinforce what we already know.

That feels productive, but it can make the surrounding landscape invisible.

Deliberate noise might mean:

  • Reading outside your profession
  • Working with people whose assumptions differ from yours
  • Traveling without optimizing every hour
  • Building something with no obvious commercial value
  • Following a curiosity before you can justify it
  • Allowing a project to fail after it has taught you enough

Not every detour will lead somewhere valuable. That is the point. Exploration cannot be fully optimized in advance because its value comes from discovering what you did not know to seek.

Know when to stay

There is a danger in turning exploration into an identity.

If you continually restart, you never experience compounding. You remain a sophisticated beginner: good at entering fields, poor at producing within them.

At some point, the evidence becomes strong enough to commit. You find a valley where your curiosity, ability, relationships, and opportunity reinforce one another. Then the strategy changes.

You reduce the learning rate. You tolerate repetition. You remain after the initial excitement disappears. You give accumulated knowledge enough time to become judgment.

Exploration helps you select the game. Commitment allows you to become good at it.

The challenge is knowing when to switch.

A useful signal is not simply enjoyment. Many worthwhile pursuits become frustrating once novelty fades. A better signal is whether deeper involvement keeps revealing more depth. Some subjects become smaller as you understand them. Others expand.

Stay with the ones that expand.

Build a portfolio of descents

We tend to imagine life as a single optimization problem: choose the right career, discover a purpose, and follow one path to its conclusion.

It may be more useful to think in terms of a portfolio.

You can descend into several valleys across different parts of your life. One may concern your work, another your family, another your health, and another an intellectual obsession with no practical purpose.

You can also revisit old terrain with new capabilities. A field you abandoned ten years ago may become newly valuable when combined with something learned since.

Nothing has to be wasted. Explorations that appear disconnected can later become the basis of a distinctive point of view.

The landscape changes

Unlike a mathematical function, life does not offer a fixed landscape.

Technology changes it. Relationships change it. Children change it. Age changes it. Your own movement alters what becomes possible next.

This means there may be no permanent global minimum to find. The best available direction can change while you are pursuing it.

The goal, then, is not to locate one perfect answer and remain there forever. It is to develop a process for moving through uncertainty:

  • Explore widely enough to see multiple valleys.
  • Go deeply enough to understand their shape.
  • Commit long enough to benefit from compounding.
  • Add enough noise to escape comfortable traps.
  • Periodically climb high enough to look at the landscape again.

Gradient descent can help us improve a known answer. A good life also requires deciding which answers are worth optimizing.