Our backend service carries several kinds of traffic at once:
Some user requests trigger external operations that are slow and expensive, such as creating Merge Requests and running pipelines.
In a service like this, the cross-cutting concerns themselves (rate limiting, feature switches, authorization, idempotency, metrics) can also go wrong: configuration can be missing or wrong, dependencies can misbehave, and instrumentation code can throw. Our goals are:
Map directly, so cross-cutting logic cannot depend on a common return type.Interceptors are registered in the following order, and Spring executes them in registration order:
@RateLimit.Rate limiting comes first for two reasons. First, rejected requests should not consume any downstream processing. Second, business activity counts measure “how many business actions users completed”, and requests rejected with 429 should not be included; they are counted by a separate “rate-limited” metric.
Business activity metrics use a path allowlist rather than wildcards, so traffic unrelated to that business never enters this interceptor at all.
Implementing rate limiting and authorization in interceptors, which run before argument binding, has two benefits:
@PostMapping("/apply")
@RateLimit(bucket = "apply", limitKey = "APPLY_RATE_LIMIT", defaultLimit = 10,
message = "Too many submissions, please try again later")
public ResponseEntity<?> apply(@RequestBody Map<String, String> body) { ... }
public boolean preHandle(HttpServletRequest req, HttpServletResponse resp, Object handler) {
RateLimit rl = annotationOf(handler);
if (rl == null) return true;
int limit = config.getInt(rl.limitKey(), rl.defaultLimit()); // Read on every request, so hot reloads take effect immediately
if (limit <= 0) return true; // Cleared or non-positive: stop rate limiting, rather than blocking everything
String subject = resolveSubject(req);
if (limiter.tryAcquire(rl.bucket() + ":" + subject, limit, windowMillis())) return true;
metrics.recordRateLimited(subject, rl.bucket());
writeJson(resp, 429, ApiResponse.error("RATE_LIMITED", rl.message()));
return false;
}
The rate-limit subject prefers the login identity. That way the same person switching browsers or opening several tabs consumes the same quota. Only when no identity can be read do we fall back to the client address, prefixed with ip: to distinguish it from identities. We fall back to the address rather than a shared bucket because any single caller could exhaust a shared bucket, affecting every caller whose identity cannot be determined.
The limiter is a sliding-window log. Each bucket keeps a queue of timestamps, and “evict expired timestamps, check, append” happens inside ConcurrentHashMap.compute, so the whole operation is atomic per bucket. When a bucket becomes empty it returns null, and the key is removed:
buckets.compute(key, (k, q) -> {
Deque<Long> d = (q == null) ? new ArrayDeque<>() : q;
while (!d.isEmpty() && now - d.peekFirst() >= windowMillis) d.pollFirst();
if (d.size() < limit) { d.addLast(now); allowed[0] = true; }
return d.isEmpty() ? null : d;
});
Trade-offs belong in the comments. Rate limiting runs before the idempotency check, so a duplicate submission that exceeds the quota gets a 429 rather than “duplicate submission”. We accept this and state it in the endpoint’s comments, so that future maintainers do not mistake it for a bug.
Hot-reloaded configuration can go missing or be wrong at any time. We explicitly specify the fallback behavior for every configuration item, with this rule: look at the main consequence of misconfiguration, and default toward the less harmful outcome.
| Configuration item | Main consequence of misconfiguration | Default behavior |
|---|---|---|
| Rate-limit threshold | Blocking legitimate traffic | No rate limiting (a threshold ≤ 0 disables rate limiting) |
| Access mode of a restricted resource | The resource is opened unintentionally | Unknown values are treated as “closed” |
| Access time window of a restricted resource | The resource is opened unintentionally | Any missing or malformed field means “outside the window” |
| Rule codes required for authorization | Unauthorized access | Missing means deny |
| New-workflow switch | Enabling an unverified path | Use the old workflow |
| Retry count, backoff time | Negative values cause skipped calls or negative sleeps | Clamped to a lower bound of 0 |
The access mode is parsed as follows:
AccessMode resolve(String raw) {
if (raw == null || raw.isBlank()) return TEST_WINDOW; // Not configured: restricted access (the default)
return switch (raw.trim().toUpperCase(Locale.ROOT)) {
case "OPEN" -> OPEN;
case "TEST_WINDOW" -> TEST_WINDOW;
default -> CLOSED; // Typos or any other unknown value: closed
};
}
The access time window is parsed in a fixed time zone rather than the JVM’s default, so the same configuration maps to the same period of time on every host.
__authenticatedOperator parameter with the login identity. The order must not be reversed; otherwise a client could forge the operator by submitting a parameter with the same name. Audit logs record only parameters that have been masked and length-truncated.At the API layer we distinguish two kinds of errors:
// Client error: return an actionable hint; the details go only to the log
ResponseEntity<?> clientFacingError(String code, String userMessage, String logDetail) {
log.warn("rejected code={} detail={}", code, logDetail);
return ApiResponse.error(code, userMessage);
}
// Server error: the log records the cause and stack trace; the client receives only a generic message
ResponseEntity<?> internalError(String code, String logDetail, Throwable cause) {
log.error("internal code={} detail={}", code, logDetail, cause);
return ApiResponse.error(code, GENERIC_ERROR_MESSAGE);
}
Status codes are designed around “what the client should do next”, not “what happened inside the server”:
DUPLICATE_SUBMIT is separate from ACCEPTED. Both have HTTP status 202, but when the frontend receives DUPLICATE_SUBMIT it keeps the form locked instead of treating it as a newly accepted request.Idempotency. Write endpoints use the client-supplied operationId as the idempotency key, relying on a database unique constraint to “look up, insert, and re-read on conflict” (see Section 3.4 of the database post). Among concurrent duplicate requests, only one proceeds; the others return the winner’s result.
Long-running operations. A long-running operation returns HTTP 202 after synchronous validation and then executes in a bounded thread pool. When the pool rejects a task, the failure state is written to the database (see Section 3.2 of the MR lifecycle post). The client can always see an explicit state by polling.
We set a few rules for metrics code and enforced each one in the implementation:
preHandle always returns true and never writes a response.request.getParameter rather than reading the request body itself, which would consume the input stream and break later argument binding.try/catch, and exceptions are only logged. Tag values are normalized and truncated to 128 characters; empty or invalid dimensions cause the emission to be skipped.afterCompletion, and only for 2xx responses.public void incrementUserCounter(String name, Map<String, String> tags) {
try {
Map<String, String> safe = normalizeAndTruncate(tags, 128);
if (safe == null) { log.warn("skip metric {}: invalid tags", name); return; }
registry.counter(name, safe).inc();
} catch (RuntimeException e) {
log.error("metric {} failed; business request continues", name, e); // A metrics failure does not affect the business request
}
}
After an SPA release, already-open old pages request hashed assets that were removed in the new version. Without handling, these requests fall through to the global exception handler and return 500. We address this on both the server and the client:
Server side:
index.html is served with Cache-Control: no-store, so the latest asset manifest is always fetched;max-age=365d, public, immutable;Accept: text/html with no file extension in the path) get a fallback page, while static asset or API requests get a 404 JSON. Both responses set no-store.Client side:
sessionStorage. Within 30 seconds, do not reload automatically again; show a “Refresh page” button instead, to avoid a reload loop.We also wrote smoke scripts for these response headers and status codes and run them in the pre-release environment.
X-Forwarded-For, which clients can forge. We plan to use the hop appended by a trusted proxy instead, or to rate-limit uniformly at the gateway.operationId and does not replay the original response. If the client does not supply an operationId, the server-generated ID contains a timestamp, so retried requests cannot be deduplicated.try/catch. Whether it truly cannot affect business logic depends on the tracing and metrics libraries not throwing. In addition, some user endpoints are outside the coverage of the API metrics.Our system applies users’ change requests by opening Merge Requests (MRs) against a Git repository. If every request opened its own MR, many MRs would modify the same repository at the same time, force each other to rebase, and queue up on the pipelines. To avoid this we introduced batch aggregation: once a request is approved it enters a waiting set, and each round merges the waiting requests into a single MR that is processed as a unit.
This design requires the following invariants to hold at all times:
The goal is to have the database enforce these invariants directly, rather than relying on “application code that happens to be correct”.
CREATE UNIQUE INDEX ... WHERE status = 'RUNNING').CREATE TABLE merge_batch (
id BIGINT NOT NULL AUTO_INCREMENT,
status TINYINT NOT NULL DEFAULT 1 COMMENT '1=RUNNING 2=SUCCEEDED 3=FAILED 4=BLOCKED',
active_slot TINYINT GENERATED ALWAYS AS
(CASE WHEN status IN (1, 4) THEN 1 ELSE NULL END) STORED,
mr_id VARCHAR(64) NULL,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (id),
UNIQUE KEY uk_active_slot (active_slot),
KEY idx_status_created (status, created_at, id)
) ENGINE = InnoDB;
This design relies on one semantic of MySQL unique indexes: a unique index allows multiple NULLs.
active_slot = 1, and the unique index guarantees that at most one row in the whole table has the value 1.active_slot = NULL, so any number of them can coexist.UPDATE status that moves a batch into a terminal state also turns active_slot into NULL, releasing the slot with it; no separate release step is needed.In effect this is equivalent to the partial unique index UNIQUE (status) WHERE status IN (RUNNING, BLOCKED).
We also compared other mutual-exclusion approaches:
| Approach | Main problem |
|---|---|
A lock row plus SELECT ... FOR UPDATE |
The lock is either held for the whole batch (a transaction lasting minutes), or a separate lease field has to be maintained |
Session-level GET_LOCK() |
The lock is bound to a database connection, which does not fit well with connection reuse in a pool |
| An external distributed lock such as Redis | Requires a new component; the lease expiry is hard to align with the task’s actual duration |
| A generated column plus a unique index | The lock and the state are the same column: there is no lease, and “state is terminal but the lock was not released” cannot happen |
The last row is why we chose it: the mutual-exclusion state is the business state itself. If the holder crashes, the batch stays RUNNING and the slot remains occupied, but this is an explicit state that can be queried and handled, not an invisible leaked lock. A timeout sweep or the operations tool moves it to a terminal state, and the slot is released. The blocked state (BLOCKED) also occupies the slot, so subsequent batches stop until this batch has been dealt with.
CREATE TABLE merge_batch_request (
id BIGINT NOT NULL AUTO_INCREMENT,
operation_id VARCHAR(36) NOT NULL,
batch_id BIGINT NULL COMMENT 'NULL means still waiting',
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (id),
UNIQUE KEY uk_operation_id (operation_id),
KEY idx_batch_created (batch_id, created_at, id)
) ENGINE = InnoDB;
Each round’s claim is done in a single transaction:
@Transactional
public Claim claim(int maxRequests) {
// 1. Take the slot
long batchId;
try {
batchId = insertReturningKey("INSERT INTO merge_batch (status) VALUES (1)");
} catch (DuplicateKeyException e) {
return Claim.slotOccupied(); // This transaction has not written anything yet; returning normally is equivalent to rolling back
}
// 2. Read the waiting requests: a plain read, no FOR UPDATE
List<Req> pending = query(
"SELECT id, operation_id FROM merge_batch_request " +
"WHERE batch_id IS NULL AND created_at < CURRENT_TIMESTAMP " +
"ORDER BY created_at, id LIMIT ?", maxRequests);
if (pending.isEmpty()) {
throw new NothingToClaimException(); // Roll back, releasing the slot just taken
}
// 3. Guarded claim: only claim rows that are still unclaimed, and the affected row count must match the number of rows read
int updated = update(
"UPDATE merge_batch_request SET batch_id = ? " +
"WHERE batch_id IS NULL AND id IN (" + placeholders(pending) + ")",
batchId, ids(pending));
if (updated != pending.size()) {
throw new ClaimPreemptedException(); // Some row was changed elsewhere; roll back everything
}
// 4. Write the append-only membership history
insertHistory(batchId, pending);
return Claim.claimed(batchId, pending);
}
Why step 2 does not use FOR UPDATE. Under the REPEATABLE READ isolation level, a locking read with a range condition takes next-key locks on the scanned index range. Here the scanned range is the batch_id IS NULL range of idx_batch_created, which is exactly where newly admitted requests are inserted, so they would be blocked until the claim transaction ends, violating I3. Dropping the locking read does not affect correctness:
WHERE batch_id IS NULL and the affected-row-count check in step 3 detect it and roll back the whole transaction.Why an empty read throws an exception. If the transaction committed normally, it would leave behind a RUNNING batch with no members that still occupies the slot. Throwing makes the slot roll back together with the transaction.
A lock-free pre-check outside the transaction. Before entering the claim transaction, we first run SELECT ... WHERE batch_id IS NULL ... LIMIT 1. When the waiting set is empty, the round ends immediately, avoiding the idle churn of “take the slot, then roll back”.
Use the database clock throughout. The condition is written as created_at < CURRENT_TIMESTAMP, and the timeout sweep as created_at < DATE_SUB(CURRENT_TIMESTAMP, INTERVAL ? SECOND). The write time and the comparison time come from the same clock, so clock skew between application servers does not affect the decision.
One thing to verify in your implementation. Step 1 catches the unique-key conflict inside the transaction and returns normally. In MySQL, a failed statement rolls back only that statement, and the transaction itself remains usable. However, some data-access frameworks mark the transaction as rollback-only when they translate the exception, causing an error on commit. Before adopting this pattern, confirm how your framework behaves.
Every state write includes the expected current state and checks the affected row count:
-- Batch terminal transition: only a batch that is still RUNNING can transition
UPDATE merge_batch SET status = 2 WHERE id = ? AND status = 1; -- Affected rows must be 1
-- Optimistic concurrency control for ordinary rows: the state just read serves as the version
UPDATE operation SET status = ?, ... WHERE operation_id = ? AND status = ?;
An affected row count of 0 means the state has been changed elsewhere, and the writer throws a “preempted” exception.
The position of such a guarded write inside a composite transaction also matters. The settlement transaction for a failed batch has several steps: mark the batch as FAILED, then delete or reset the records of each request in the batch. We put the batch’s guarded write first:
tx.execute(List.of(
() -> batchRepo.markFailed(batchId, code, msg), // WHERE status = RUNNING; affected rows must be 1
() -> resetOperations(batchId) // Subsequent steps
));
If the batch has already been settled by another path (for example, the timeout sweep and the running task arrive at the same time), the first step fails, the whole transaction rolls back, and the reset does not run a second time.
A unique-constraint conflict has a clear business meaning in each scenario, and we handle each one accordingly:
// Admission: a duplicate means "already admitted"; return false, no error
public boolean admit(String opId) {
try {
insert("INSERT INTO merge_batch_request (operation_id) VALUES (?)", opId);
return true;
} catch (DuplicateKeyException e) {
return false;
}
}
// Audit record: a duplicate means "already recorded"; do not overwrite the existing row (it may already be terminal)
public void insertAudit(Audit a) {
try { insert(a); } catch (DuplicateKeyException e) { /* Expected deduplication result */ }
}
// Request idempotency: look up first; insert if not found; an insert conflict means a concurrent request inserted first, so re-read and return the winner's result
public Result checkOrInsert(String requestId, String opId) {
Record r = findById(requestId);
if (r != null) return Result.duplicate(r.operationId());
try {
insert(new Record(requestId, opId));
return Result.fresh();
} catch (DuplicateKeyException e) {
Record winner = findById(requestId);
return winner != null ? Result.duplicate(winner.operationId())
: Result.retryLater();
}
}
None of these three patterns takes an explicit lock; their correctness under concurrency comes from the unique constraint itself.
The binding table has a unique key on (subject_id, target_key), so each relationship has exactly one row. But every request generates a new operation_id, so the “one row per operation” write model has to be reconciled with the “one row per relationship” unique key:
Binding existing = findBySubjectAndTarget(subject, target);
if (existing == null) {
insert(newRow); // First application
} else if (REAPPLIABLE.contains(existing.status())) {
int n = update(
"UPDATE binding SET status = ?, operation_id = ?, expires_at = ? " +
"WHERE subject_id = ? AND target_key = ? AND status = ?", // Pin the exact old state that was read
newRow.status(), newRow.operationId(), newRow.expiresAt(),
subject, target, existing.status());
if (n == 0) throw new PreemptedException(); // A concurrent re-application or state change
} else {
throw new ConstraintViolationException(); // Refuse to overwrite an active row or one awaiting recovery
}
operation_id, which is exactly how stale callbacks are recognized in the previous post.operation_id would be replaced, and the recovery flow attached to the old operation_id would no longer find its target.Transitions that modify both the operation and binding tables always lock operation first and binding second. All paths use the same lock order, which avoids deadlocks. The advancing method returns a non-null result only when this call actually changed the state, and callers use that to decide whether to produce side effects (emitting metrics, sending notifications). As a result, of two concurrent paths, only one produces side effects (see Section 3.7 of the previous post for details).
When a batch fails, the MR may already exist, and it may happen to merge successfully just after the failure was decided. The settlement flow is: if the MR is still open, close it first, then re-read the MR’s final state.
version column that is not yet in use. For states that may round-trip, we plan to switch to a monotonically increasing version number.CASE WHEN status IN (1, 4) THEN partition_key END, and the unique index becomes “at most one active batch per partition”.IN lists can reach over a thousand placeholders without being chunked. We plan to switch to batch queries and batch inserts, and to chunk the IN lists.status plus a time range, but only a single-column time index exists; fetching the largest MR number uses ORDER BY CAST(mr_id AS UNSIGNED), and sorting on an expression cannot use an index. We plan to add a composite (status, updated_at) index and store the MR number in a numeric column.created_at has second precision and the claim condition is created_at < CURRENT_TIMESTAMP, so requests admitted within the current second wait until the next round. This is a deliberate trade-off, but it needs to be documented.We maintain a self-service change system built on “configuration as code”. After a user submits a change request, the system has to complete the following steps in order:
A single change spans three external systems (the approval system, the Git hosting platform GitLab, and CI) and takes anywhere from a few minutes to a few hours. Handling it as an ordinary synchronous request runs into several problems: an HTTP connection cannot wait that long; a failure at any step can leave the request stuck in an intermediate state nobody knows about; callbacks from external systems may be duplicated, late, or out of order; and a process restart loses whatever step was executing.
We set four goals for this workflow:
External calls are outside the local transaction. After creating the branch, commit and MR on GitLab, the local database write can still fail; conversely, once local state is written, the remote call can still fail. There is no atomicity between the two sides to rely on.
Many event sources, with weak delivery semantics. State changes are driven by user submissions, approval callbacks, the system’s own progression, deployment pipeline callbacks, scheduled reconciliation, and manual recovery by operators. External callbacks are delivered “at least once”; they may arrive late, or even before our own state write lands.
Upstream responses are ambiguous. GitLab returns HTTP 401 when the same account approves twice, which is the same status code as “this account is not allowed to approve”. The response alone cannot distinguish the two.
Deployment topology. The service runs as multiple instances. We did not want to introduce new components such as a workflow engine or distributed locks for this one workflow; the only shared coordination point we have is the relational database.
The workflow must evolve. The step order needed to change from “create the MR, then approve” to “approve, then create the MR”, with approvals already in flight at switchover time. The upstream platform also planned to add a required structured field to its approval API, with a compatibility period.
We record state at two granularities in two tables:
operation table records step state, down to details such as “pipeline timed out”, “rebase conflict” or “bot approval failed”. Status codes are allocated in ranges by phase (creation from 10001, handling from 10101, closing from 10201), so metrics can be aggregated by prefix.binding table records external state, expressing only the phases users care about: approval pending, MR being created, merging, merged, in effect, failed, rejected. The table has a unique key on (subject_id, target_key), so each “subject–target” relationship has exactly one row.enum StepStatus { // Fine-grained: for troubleshooting and metrics
CREATE_STARTED(10001), CREATE_SUCCEEDED(10002), CREATE_PIPELINE_FAILED(10004),
HANDLE_STARTED(10101), HANDLE_APPROVED(10102), HANDLE_REBASE_CONFLICT(10105),
HANDLE_PIPELINE_PENDING(10501), CALLED_BACK(10301), /* ... */;
}
enum BindingStatus { // Coarse-grained: for display and "can the user re-apply"
APPROVAL_SENDING, APPROVAL_SENT, MR_CREATING, MR_CREATED, MERGING, MERGED, VALIDATED,
CREATE_FAILED, HANDLE_FAILED, REJECTED, CLOSED;
/** Single source of truth: both the data layer and the API layer use it to decide whether re-applying is allowed. */
static final Set<BindingStatus> REAPPLIABLE = Set.of(REJECTED, CLOSED);
}
We split state into two layers because the two audiences need different things. Troubleshooting needs to know exactly which step a request stopped at; users, and the “can I re-apply” decision, only care about the broad phase. With a single status field, every refinement of the internal steps would ripple into the frontend display and the re-apply rules.
Every state transition writes both tables in the same local transaction:
void transition(OperationBuilder op, StepStatus step, BindingStatus status) {
tx.execute(List.of(
() -> operationRepo.update(op.status(step)), // Optimistic concurrency check against the state just read
() -> bindingRepo.updateStatus(op.id(), status)
));
}
The diagram below shows the main path of the external state (failure branches omitted):
stateDiagram-v2
[*] --> APPROVAL_SENT: Request submitted
APPROVAL_SENT --> REJECTED: Approval rejected or withdrawn
APPROVAL_SENT --> MR_CREATING: Approval granted
MR_CREATING --> MR_CREATED: Check pipeline passed
MR_CREATING --> CREATE_FAILED
MR_CREATED --> MERGING
MERGING --> MERGED: Merge succeeded
MERGING --> HANDLE_FAILED
MERGED --> VALIDATED: Deploy callback or scheduled reconciliation
REJECTED --> [*]
VALIDATED --> [*]
The submit endpoint performs only synchronous validation (parameters, idempotency, whether a request is already in progress), then hands the long-running workflow to a bounded thread pool and immediately returns HTTP 202 with an operationId. The client polls by operationId.
private void fireAsync(String opId, Task task, StepStatus failStep, BindingStatus failStatus) {
try {
executor.execute(() -> {
try {
task.run();
} catch (Exception e) {
log.error("async step failed", e); // The failure state has already been written by the aspect; see below
} catch (Throwable t) {
log.error("[CRITICAL] JVM-level error", t); // The JVM may be in a bad state; log only, do not write to the DB
}
});
} catch (RejectedExecutionException e) {
// Pool is full or shutting down: persist the failure so pollers see "failed" rather than "processing"
tx.execute(List.of(
() -> operationRepo.updateStatus(opId, failStep),
() -> bindingRepo.updateStatus(opId, failStatus)));
}
}
The pool’s core size equals its maximum size, the queue is bounded, and the rejection policy is AbortPolicy. We did not choose the two other common options, for these reasons:
CallerRunsPolicy would make a web thread run a task lasting several minutes, propagating overload to every endpoint.AbortPolicy combined with “persist on rejection” turns overload into a failure state that is visible to users and can be counted.
Each step method declares its own failure state through an annotation, and an aspect provides a uniform safety net:
@WorkflowStep(onErrorStep = StepStatus.HANDLE_MERGE_FAILED,
onErrorStatus = BindingStatus.HANDLE_FAILED)
public Result handleMergeRequest(String opId, long mrIid) { ... }
Failures recognized inside the method (such as pipeline failure or rebase conflict) are written by business code with a more specific state; unexpected exceptions are written by the aspect with the state declared in the annotation. Together they guarantee that every exit path leaves a terminal state behind.
Approval callbacks are handled in two layers. The first layer deduplicates by the request ID carried in the callback itself, blocking duplicate deliveries. The second layer decides whether the callback is still valid:
Binding row = bindingRepo.findByOperationId(callback.operationId());
if (row == null || !AWAITING_APPROVAL.contains(row.status())) {
return accepted("IGNORED_STALE"); // Return 2xx so the upstream does not treat it as a failure and keep retrying
}
There are two design points here.
binding row is reset in place and given a new operationId (see the database post). A late callback for the old request therefore finds no row and is ignored, instead of pushing the new request into the wrong state.AWAITING_APPROVAL = {APPROVAL_SENDING, APPROVAL_SENT}. We write SENDING before calling the approval API and SENT after the call succeeds. The approval system’s callback can arrive before SENT is written; if the waiting set contained only SENT, such callbacks would be misjudged as stale.Because local and remote state are not atomic, every step that modifies remote state first reads the remote state before acting:
The bot approval step needs extra handling for the 401 ambiguity:
void approveAsReviewer(Api reviewer, long mrIid) throws ApiException, InterruptedException {
int maxRetries = Math.max(0, config.getInt("approval.maxRetries", 3)); // Negative values are treated as 0
ApiException last = null;
for (int attempt = 1; attempt <= maxRetries + 1; attempt++) {
if (hasApprovalFrom(reviewer, mrIid)) {
return; // Already approved: skip the call
}
try {
approver.approve(reviewer, mrIid);
return;
} catch (ApiException e) {
last = e;
if (attempt > maxRetries) break;
Thread.sleep(backoffBaseMs << (attempt - 1)); // Exponential backoff
}
}
// Both duplicate approval and unauthorized approval return 401. Read once more to check whether the approval actually exists.
if (hasApprovalFrom(reviewer, mrIid)) {
return;
}
throw last;
}
These checks make “re-run from where it stopped” safe, so the operations recovery tool can simply re-trigger a given step.
Both the rebase result and the pipeline status have to be polled. We abstracted polling into a generic component: short intervals for the first N attempts, longer intervals afterward, and a timeout once the limit is reached. A poll has three possible outcomes: passed, failed, timed out.
The API layer uses this to show users different hints: a deterministic failure says “contact the maintainers”, while an unknown outcome says “you can keep waiting” (see the API post for details).
When we changed the workflow from “create the MR, then approve” to “approve, then create the MR”, we did not schedule downtime or a drain window. Instead, both orders coexisted for a period of time:
Callbacks identify the workflow by payload shape. In the old workflow, the MR already exists when the approval is submitted, and its number is put into an extension field of the approval form; in the new workflow, there is no MR yet when the approval is submitted. Callback handling decides accordingly:
boolean legacyFlow = StringUtils.isNotEmpty(callback.ext().get("mrId"));
if (approved) {
if (legacyFlow) workflow.fireHandle(opId, mrId); // MR already exists: approve and merge
else workflow.fireCreateAfterApproval(opId); // Create the MR first
}
Approvals sent before the switch was flipped but called back afterward are still handled by the old workflow.
The create step queries the database to decide between “insert” and “advance”, rather than relying on a flag passed by the caller:
if (operationRepo.exists(opId)) {
transition(op, CREATE_STARTED, MR_CREATING); // New workflow: the row was created when the approval was submitted
} else {
insertOperationAndBinding(op); // Old workflow: first-time creation
}
This makes the same step valid for both orders, and keeps it idempotent when operators re-trigger it.
Callbacks from the deployment pipeline can be lost, leaving the corresponding requests stuck at “merged”. A scheduled job converges such records.
Sliding time window. Each run scans only “merged” records whose last-modified time falls within (now - maxAge, now - minAge). Records newer than minAge may simply have their callback still on the way, so they are skipped; records older than maxAge fall out of the window, which bounds the work per run.
A watermark instead of per-row lookups. The callback table records only MR numbers, with no business fields. The reconciliation job runs just two queries: one IN query that fetches operationId → mrIid in bulk, and one that fetches the largest MR number in the callback table. Then it does one comparison per row:
mrIid <= maxCallbackIid → advance to "in effect"
mrIid > maxCallbackIid → stay "merged", wait for the next run
mrIid == max means this MR’s own callback has arrived; mrIid < max means an MR with a larger number has already been called back, and since MR numbers increase monotonically within a project, we infer that the callback for this earlier MR was lost. This inference holds on the condition that the deployment pipeline runs in merge order, and a later run never skips changes merged before it. The minAge window prevents “late” from being mistaken for “lost”.
Takes effect exactly once when racing with the normal callback path. Both the callback handler and the reconciliation job call the same state-advancing method:
@Transactional
public String markValidated(String opId) {
Operation op = operationRepo.selectForUpdate(opId); // Fixed lock order: operation first
Binding b = bindingRepo.selectForUpdate(opId); // then binding
if (b == null || (b.status() != MERGED && b.status() != VALIDATED)) {
return null; // Preconditions not met: idempotent no-op
}
if (op.status() != CALLED_BACK) {
operationRepo.updateStatus(opId, CALLED_BACK); // Repair a step state that may be lagging
}
if (b.status() == MERGED) {
requireOneRow(bindingRepo.update(
"SET status = VALIDATED WHERE operation_id = ? AND status = MERGED", opId));
return b.subjectId(); // Non-null only when the state actually flipped
}
return null;
}
When the two paths race, row locks ensure only one of them completes the flip; the other reads VALIDATED and returns null. Callers count a “validated successfully” metric only when the return value is non-null, and emit it only after the outer transaction commits, so a rollback cannot leave an over-counted metric and concurrency cannot cause double counting.
The upstream platform planned to add a required field to its approval API: a list of check items that must each be acknowledged. The GitLab client library we use cannot express this field. Our approach:
A mature workflow engine provides durable execution, automatic retries and crash recovery, which are exactly the parts this design implements by hand. We did not adopt one because the workflow has a limited number of steps; the only shared infrastructure the team already has is the relational database; and the genuinely hard parts (ambiguous external responses, deciding when a callback is stale, idempotency based on remote state) are business semantics that an engine cannot solve for us. The cost is that we have to fill in some durable-execution capabilities ourselves; see Section 5.
operationId to see its phase and the exact step it stopped at; troubleshooting no longer requires reconstructing the process from logs.