🎧 The
Stream Works on Web… So Why Is Mobile Broken?

I am on call when an
uncomfortable pattern appears in our monitoring: playback requests are
healthy for most web users, but a growing group of mobile sessions is
failing shortly after playback begins. The service is still online, so this
is not a simple outage. Some clients work, some do not, and the failures
are spread across regions. That makes the incident much harder than a
server that is simply down.

The first thing I try to avoid is
guessing. A modern media product is a chain of systems. A phone or browser
calls an API. That API may contact several backend services. Those services
can read from databases, caches, or cloud storage and may return metadata
that tells a client where and how to request media. Infrastructure
configuration, deployment state, network behavior, serialization, client
versions, and protocol assumptions can all affect the final experience.

One user action can cross many systems

When a listener presses
play, the action looks simple from the outside. Internally, however, the
request may cross multiple boundaries. A client needs metadata. Backend
services need to respond quickly and consistently. Data stores need to
provide the correct state. Media delivery components need to expose usable
resources. The client then needs to interpret everything correctly.

This is why backend engineering is not only about writing an endpoint
that returns a successful response. I have to think about the clients
consuming that endpoint, the contracts between services, the infrastructure
underneath them, and what happens when one assumption changes.

The
difficult bugs are often partial failures

A total failure is
obvious. A partial failure can be deceptive. Imagine that a newly deployed
backend response contains a field that older mobile clients interpret
differently. The web application may already understand the new behavior,
while an older phone application does not. Server dashboards can still look
mostly healthy because the API technically returns successful
responses.

In that situation, I need to correlate evidence instead
of staring at one dashboard. I compare client versions, request traces,
error rates, deployment times, regions, and recent configuration changes. I
ask whether the failure began before or after a release. I check whether
one platform is affected more strongly than another. I also look for
secondary symptoms: retries, latency increases, database pressure, or
unusual traffic between services.

API design is a compatibility
problem

An API used by several client platforms has a longer life
than a single implementation. Android, iOS, web, and other consumers may
update at different speeds. That means a backend change that looks clean in
isolation can still be dangerous if it changes assumptions that existing
clients depend on.

I therefore treat compatibility as part of API
design. A change may need versioning, a migration period, a feature flag,
or a backward-compatible response. Communication patterns also matter. A
synchronous request, a typed RPC interface, and a graph-oriented API can
create different contracts and operational trade-offs.

Scaling is
not the same as adding machines

Serving a large audience requires
more than increasing instance counts. I need to understand where work
accumulates. Is the database receiving an inefficient query pattern? Is a
service repeatedly requesting the same information? Is a cache ineffective?
Are retries amplifying a small failure? Is one dependency becoming a
bottleneck?

Good backend systems are designed so engineers can
reason about these questions. Clear service boundaries, useful metrics,
traces, structured logs, sensible timeouts, and defensive failure handling
make incidents easier to understand. Automated tests and continuous
delivery reduce risk, but they are most valuable when combined with careful
engineering judgment.

Infrastructure is part of the
application

Cloud resources and infrastructure configuration are
not separate from software behavior. A change to networking, permissions,
autoscaling, container configuration, or a managed database can alter how
an application behaves in production. Infrastructure as code helps because
changes can be reviewed, reproduced, and tracked instead of existing only
as manual console actions.

During an incident, I want to know not
only which application commit changed, but also whether infrastructure
changed at the same time. A small configuration difference between
environments can explain why a bug is invisible in testing but obvious
under production traffic.

On-call work is controlled
decision-making

When users are affected, speed matters, but
uncontrolled speed creates new failures. I need to decide whether to roll
back, disable a feature, reduce traffic to a dependency, or deploy a
targeted fix. The safest action is not always the most sophisticated one.
Sometimes restoring compatibility quickly is more valuable than proving the
exact root cause before taking action.

Ownership also continues
after recovery. Once playback is stable, I still need to understand why the
problem escaped testing, why monitoring did or did not detect it early, and
what would prevent the same class of failure from returning.

Engineering is collaborative even when the bug is in the backend

A backend engineer cannot solve every cross-platform problem alone.
Mobile engineers understand client release patterns and platform-specific
behavior. Web engineers can confirm whether the same API contract behaves
differently in browsers. Infrastructure specialists can help interpret
cloud and deployment changes. Code review provides another perspective
before risky changes reach production.

The strongest incident
response is therefore not a heroic individual debugging everything in
isolation. It is a team building a shared model of the failure, testing
hypotheses against evidence, communicating clearly, and choosing the
lowest-risk path back to a reliable user experience.

From fixing
one incident to improving the system

After recovery, I want the
post-incident discussion to go beyond “someone introduced a bug.” A useful
review asks which technical and organizational conditions allowed the
failure to happen. Did an API contract change without sufficient
compatibility testing? Did automated tests cover the new server behavior
but not older clients? Was a rollout too broad? Were alerts based only on
server errors instead of playback success?

At an advanced level,
backend engineering is the practice of making these trade-offs explicit.
Performance, scalability, maintainability, reliability, delivery speed, and
compatibility can pull in different directions. Good engineering means
understanding those tensions and making decisions that the team can
explain, review, operate, and improve.

The German conversations in
this lesson use the same production incident at increasing levels of
linguistic and technical complexity. At the beginning, I report that
playback is broken and ask for help. Later, I explain evidence, challenge
assumptions, coordinate rollback decisions, discuss service boundaries and
compatibility, and finally evaluate architectural and operational lessons
from the incident.