Cloud Scheduler & Cloud Tasks: Notification System

The Problem and Why It Mattered

I designed and implemented a production notification and background job system on GCP — Cloud Scheduler for time-triggered orchestration and Cloud Tasks for deferred, retriable, rate-limited execution — replacing a brittle cron-on-VM setup. The relevant technical and business risk is the cost of leaving this workflow, control, or product boundary unclear, manual, insecure, or difficult to operate. The case study focuses on the decisions used to make that risk manageable.

Objectives and Success Criteria

• Deliver the capability described in the project objective using a maintainable technical design.
• Make the main workflow, control boundary, and operational responsibilities explicit.
• Produce reviewable implementation evidence without publishing confidential values or customer data.

Requirements and Constraints

The implementation had to be reproducible, observable, and safe to describe publicly without exposing operational secrets or private infrastructure detail.

My Role and Ownership

I owned the technical design and implementation described in this portfolio record, including the technology choices, core system boundaries, and the documentation of the resulting approach. Related organisational decisions, vendor operations, and formal certification decisions remain outside the scope of this technical case study unless stated explicitly.

Solution Overview

The solution is built around Cloud Scheduler, Cloud Tasks, Cloud Run. Its implementation is organised as the components below so that the architecture, control points, and operating model can be evaluated separately rather than treated as a list of tools.

Architecture and System Design

The public architecture description follows the path from the initiating user, workload, or operator action through the relevant cloud engineering components. Access boundaries, data or state handling, and operational controls are described at a safe level; secrets, private addresses, and client-sensitive implementation details are intentionally omitted.

Technology Decisions and Trade-Offs

I selected Cloud Scheduler, Cloud Tasks, Cloud Run because they support the project objective and make the critical engineering boundaries explicit. The trade-off is operational and platform complexity: the design depends on documented configuration, controlled access, and ongoing review rather than assuming that a managed service or framework removes engineering responsibility.

Implementation Walkthrough

The following implementation notes describe the components that matter most to the project outcome. They are intentionally focused on technical decisions and operating behaviour instead of reproducing entire source files or sensitive configuration.

Implementation: Architecture

Cloud Scheduler publishes trigger messages to Pub/Sub topics on a defined schedule. Cloud Run services subscribe and perform orchestration, then create one Cloud Tasks task per user. This decouples trigger from execution.

Implementation: Cloud Tasks — Rate Limiting, Retries & Deduplication

Each notification is a discrete Cloud Tasks task targeting a Cloud Run handler. Cloud Tasks provides rate limiting (500 tasks/sec), automatic retries with exponential backoff (5 retries), deduplication by task name, and scheduling delay.

Implementation: Notification Types

Document expiry reminders (daily 06:00 WAT), weekly digest emails (Monday 07:30 WAT), push notifications via Firebase Cloud Messaging, and webhook delivery to enterprise customers with retry logic for non-2xx responses up to 24 hours.

Implementation: Dead-Letter Handling

Tasks that exhaust all retries publish to a dead-letter Pub/Sub topic. A Cloud Function consumes this topic, logs the failure to BigQuery, and sends a Slack alert — no failed notification goes unnoticed.

Implementation: Observability

Cloud Tasks queue metrics feed a Cloud Monitoring dashboard alongside Cloud Run handler metrics. SLO alerts: if task failure rate exceeds 1% or queue depth exceeds 10,000 pending tasks, PagerDuty is notified.

Implementation: Results

Eliminated 100% of silent notification failures caused by VM maintenance events. Consistent notification delivery within ±2 minutes of scheduled time. Rate limiting reduced SendGrid API errors from ~200/month to zero.

Challenges, Failures and Corrections

This portfolio record does not invent deployment failures, client incidents, or corrective actions that are not supported by the published project evidence. Where operational details are confidential or were not captured for public release, the case study documents the design boundary and lesson instead of presenting an unsupported narrative.

Security, Compliance and Risk

The public record avoids secrets, customer data, and private infrastructure details. The relevant security controls, access boundaries, and operational risks are addressed in the implementation notes where they are material to the project.

Testing, Validation and Evidence

The evidence published for this project is the documented implementation, its stated architecture and controls, and the portfolio record. Eliminated 100% of silent notification failures caused by VM maintenance events. Consistent notification delivery within ±2 minutes of scheduled time. Rate limiting reduced SendGrid API errors from ~200/month to zero. Claims are limited to the implementation details and outcomes documented in this case study.

Results and Evidence

Eliminated 100% of silent notification failures caused by VM maintenance events. Consistent notification delivery within ±2 minutes of scheduled time. Rate limiting reduced SendGrid API errors from ~200/month to zero.

What I Learned

This project reinforced that cloud engineering work is not a technology checklist. The durable value comes from making boundaries, trade-offs, ownership, and evidence clear enough for another engineer or stakeholder to review and operate the system responsibly.

What I Would Improve Next

• Expand automated validation around the most important workflow and control boundaries.
• Add or deepen operational evidence, such as dashboards, test artefacts, or an architecture diagram where it is safe to publish one.
• Review the design against the next expected scale, reliability, and governance requirement before extending the system.

Final Takeaway

This cloud engineering case study addresses a defined technical problem through Cloud Scheduler, Cloud Tasks, Cloud Run. It demonstrates my ability to turn requirements into an explainable design, implement the important controls, and document outcomes and limitations without overstating what the evidence proves.