Cloud Run Story Notification Job Recovery

The Problem and Why It Mattered

A production incident remediation that restored MyHealthDiary's daily Cloud Run notification job after a Docker dependency failure prevented Firebase broadcasts. The relevant technical and business risk is the cost of leaving this workflow, control, or product boundary unclear, manual, insecure, or difficult to operate. The case study focuses on the decisions used to make that risk manageable.

Objectives and Success Criteria

• Deliver the capability described in the project objective using a maintainable technical design.
• Make the main workflow, control boundary, and operational responsibilities explicit.
• Produce reviewable implementation evidence without publishing confidential values or customer data.

Requirements and Constraints

The delivery design had to be repeatable across environments, minimise direct production access, and retain enough evidence to investigate and roll back change safely.

My Role and Ownership

I owned the technical design and implementation described in this portfolio record, including the technology choices, core system boundaries, and the documentation of the resulting approach. Related organisational decisions, vendor operations, and formal certification decisions remain outside the scope of this technical case study unless stated explicitly.

Solution Overview

The solution is built around Python, Docker, Cloud Run Jobs. Its implementation is organised as the components below so that the architecture, control points, and operating model can be evaluated separately rather than treated as a list of tools.

Architecture and System Design

The public architecture description follows the path from the initiating user, workload, or operator action through the relevant devops components. Access boundaries, data or state handling, and operational controls are described at a safe level; secrets, private addresses, and client-sensitive implementation details are intentionally omitted.

Technology Decisions and Trade-Offs

I selected Python, Docker, Cloud Run Jobs because they support the project objective and make the critical engineering boundaries explicit. The trade-off is operational and platform complexity: the design depends on documented configuration, controlled access, and ongoing review rather than assuming that a managed service or framework removes engineering responsibility.

Implementation Walkthrough

The following implementation notes describe the components that matter most to the project outcome. They are intentionally focused on technical decisions and operating behaviour instead of reproducing entire source files or sensitive configuration.

Implementation: Incident diagnosis

Cloud Scheduler triggered the job correctly, but the deployed container exited at startup because its dedicated dependency file omitted firebase-admin; correcting that exposed a second missing APScheduler dependency.

Implementation: Build-time reliability control

Cloud Build now runs an import smoke test inside the actual stories image with synthetic environment values before pushing or deploying it, catching dependency drift in the deployed artefact.

Implementation: Controlled recovery and validation

The job reclaims failed or stale daily-story states under the same daily key, uses one retry instead of three, and completed a verified post-fix execution with a broadcast attempt to 368 registered users.

Challenges, Failures and Corrections

This portfolio record does not invent deployment failures, client incidents, or corrective actions that are not supported by the published project evidence. Where operational details are confidential or were not captured for public release, the case study documents the design boundary and lesson instead of presenting an unsupported narrative.

Security, Compliance and Risk

The implementation is described with least privilege, traceability, controlled configuration, and evidence boundaries in mind. Where the project supports a regulated environment, these are technical safeguards and audit-readiness contributions, not a claim that an application or product is independently certified.

Testing, Validation and Evidence

The evidence published for this project is the documented implementation, its stated architecture and controls, and the portfolio record. Restored a failed daily job, added image import validation before deployment, reduced retry amplification, and completed a verified broadcast attempt to 368 registered users. Claims are limited to the implementation details and outcomes documented in this case study.

Results and Evidence

Restored a failed daily job, added image import validation before deployment, reduced retry amplification, and completed a verified broadcast attempt to 368 registered users.

What I Learned

This project reinforced that devops work is not a technology checklist. The durable value comes from making boundaries, trade-offs, ownership, and evidence clear enough for another engineer or stakeholder to review and operate the system responsibly.

What I Would Improve Next

• Expand automated validation around the most important workflow and control boundaries.
• Add or deepen operational evidence, such as dashboards, test artefacts, or an architecture diagram where it is safe to publish one.
• Review the design against the next expected scale, reliability, and governance requirement before extending the system.

Final Takeaway

This production cloud operations and reliability remediation case study addresses a defined technical problem through Python, Docker, Cloud Run Jobs. It demonstrates my ability to turn requirements into an explainable design, implement the important controls, and document outcomes and limitations without overstating what the evidence proves.