1. Problem and Goals

Our backend service carries several kinds of traffic at once:

  • User-facing self-service endpoints: submitting requests, checking status, viewing my records;
  • Callbacks from external systems;
  • Operations endpoints;
  • A single-page application (SPA) frontend.

Some user requests trigger external operations that are slow and expensive, such as creating Merge Requests and running pipelines.

In a service like this, the cross-cutting concerns themselves (rate limiting, feature switches, authorization, idempotency, metrics) can also go wrong: configuration can be missing or wrong, dependencies can misbehave, and instrumentation code can throw. Our goals are:

  • For every cross-cutting concern, explicitly define how it behaves when “a dependency is missing, the configuration is wrong, or a dependency fails”;
  • Misconfiguration must not open features that should be closed, nor block traffic that should be allowed;
  • Metrics collection never changes business outcomes;
  • Error responses are useful to clients without leaking internal details.

2. Constraints and Considerations

  • The service runs as multiple instances, with no shared cache dedicated to rate limiting.
  • Configuration comes from a configuration center and supports hot reload. This means configuration may be missing, empty or misspelled, and can change at runtime.
  • The caller’s identity comes from a unified login SDK whose fields are not uniform. Sometimes it is an email-style account, sometimes an employee ID, and both need to resolve to the same identity.
  • Existing endpoints’ response shapes must not change. Some legacy endpoints return a Map directly, so cross-cutting logic cannot depend on a common return type.
  • Old pages linger after a frontend release. Already-open old pages request hashed static assets that were removed in the new version.
  • The team convention is that observability code reads but never writes. Instrumentation may only read values the business logic has already resolved, and must not in turn influence business logic.

3. Design

3.1 Interceptor responsibilities and order

Interceptors are registered in the following order, and Spring executes them in registration order:

  1. Rate limiting: applies only to handler methods annotated with @RateLimit.
  2. Authorization for specific paths.
  3. API latency and status-code metrics.
  4. Business activity metrics: applies only to an explicitly listed path allowlist.

Rate limiting comes first for two reasons. First, rejected requests should not consume any downstream processing. Second, business activity counts measure “how many business actions users completed”, and requests rejected with 429 should not be included; they are counted by a separate “rate-limited” metric.

Business activity metrics use a path allowlist rather than wildcards, so traffic unrelated to that business never enters this interceptor at all.

Implementing rate limiting and authorization in interceptors, which run before argument binding, has two benefits:

  • A malformed request body cannot trigger a 400 first, bypassing or pre-empting authorization;
  • The interceptor writes the response itself, so no endpoint’s return type has to change.

3.2 Annotation-driven rate limiting that runs before argument binding

@PostMapping("/apply")
@RateLimit(bucket = "apply", limitKey = "APPLY_RATE_LIMIT", defaultLimit = 10,
           message = "Too many submissions, please try again later")
public ResponseEntity<?> apply(@RequestBody Map<String, String> body) { ... }

public boolean preHandle(HttpServletRequest req, HttpServletResponse resp, Object handler) {
    RateLimit rl = annotationOf(handler);
    if (rl == null) return true;

    int limit = config.getInt(rl.limitKey(), rl.defaultLimit());   // Read on every request, so hot reloads take effect immediately
    if (limit <= 0) return true;                                    // Cleared or non-positive: stop rate limiting, rather than blocking everything

    String subject = resolveSubject(req);
    if (limiter.tryAcquire(rl.bucket() + ":" + subject, limit, windowMillis())) return true;

    metrics.recordRateLimited(subject, rl.bucket());
    writeJson(resp, 429, ApiResponse.error("RATE_LIMITED", rl.message()));
    return false;
}

The rate-limit subject prefers the login identity. That way the same person switching browsers or opening several tabs consumes the same quota. Only when no identity can be read do we fall back to the client address, prefixed with ip: to distinguish it from identities. We fall back to the address rather than a shared bucket because any single caller could exhaust a shared bucket, affecting every caller whose identity cannot be determined.

The limiter is a sliding-window log. Each bucket keeps a queue of timestamps, and “evict expired timestamps, check, append” happens inside ConcurrentHashMap.compute, so the whole operation is atomic per bucket. When a bucket becomes empty it returns null, and the key is removed:

buckets.compute(key, (k, q) -> {
    Deque<Long> d = (q == null) ? new ArrayDeque<>() : q;
    while (!d.isEmpty() && now - d.peekFirst() >= windowMillis) d.pollFirst();
    if (d.size() < limit) { d.addLast(now); allowed[0] = true; }
    return d.isEmpty() ? null : d;
});

Trade-offs belong in the comments. Rate limiting runs before the idempotency check, so a duplicate submission that exceeds the quota gets a 429 rather than “duplicate submission”. We accept this and state it in the endpoint’s comments, so that future maintainers do not mistake it for a bug.

3.3 Configuration defaults: choose the direction by the consequence of misconfiguration

Hot-reloaded configuration can go missing or be wrong at any time. We explicitly specify the fallback behavior for every configuration item, with this rule: look at the main consequence of misconfiguration, and default toward the less harmful outcome.

Configuration item Main consequence of misconfiguration Default behavior
Rate-limit threshold Blocking legitimate traffic No rate limiting (a threshold ≤ 0 disables rate limiting)
Access mode of a restricted resource The resource is opened unintentionally Unknown values are treated as “closed”
Access time window of a restricted resource The resource is opened unintentionally Any missing or malformed field means “outside the window”
Rule codes required for authorization Unauthorized access Missing means deny
New-workflow switch Enabling an unverified path Use the old workflow
Retry count, backoff time Negative values cause skipped calls or negative sleeps Clamped to a lower bound of 0

The access mode is parsed as follows:

AccessMode resolve(String raw) {
    if (raw == null || raw.isBlank()) return TEST_WINDOW;        // Not configured: restricted access (the default)
    return switch (raw.trim().toUpperCase(Locale.ROOT)) {
        case "OPEN"        -> OPEN;
        case "TEST_WINDOW" -> TEST_WINDOW;
        default            -> CLOSED;                            // Typos or any other unknown value: closed
    };
}

The access time window is parsed in a fixed time zone rather than the JVM’s default, so the same configuration maps to the same period of time on every host.

3.4 Authorization: fail closed, and reveal nothing on denial

  • Any exception from the permission service call is treated as “no permission”, and a missing rule code is likewise treated as “no permission”.
  • Multiple rule codes are checked in one round trip, for combined checks such as “administrators may also access”.
  • Distinguish 401 from 403. If no login identity can be read, or the login SDK throws, return 401; if the identity is valid but lacks permission, or the permission service fails, return 403.
  • The 403 response message is always “Access denied”, so a rejected caller cannot learn which permission is required.
  • The operator identity for the operations console is written by the server. The server first copies the parameters submitted by the client, then overwrites the reserved __authenticatedOperator parameter with the login identity. The order must not be reversed; otherwise a client could forge the operator by submitting a parameter with the same name. Audit logs record only parameters that have been masked and length-truncated.

3.5 Error responses: actionable information for clients, details stay in logs

At the API layer we distinguish two kinds of errors:

// Client error: return an actionable hint; the details go only to the log
ResponseEntity<?> clientFacingError(String code, String userMessage, String logDetail) {
    log.warn("rejected code={} detail={}", code, logDetail);
    return ApiResponse.error(code, userMessage);
}

// Server error: the log records the cause and stack trace; the client receives only a generic message
ResponseEntity<?> internalError(String code, String logDetail, Throwable cause) {
    log.error("internal code={} detail={}", code, logDetail, cause);
    return ApiResponse.error(code, GENERIC_ERROR_MESSAGE);
}

Status codes are designed around “what the client should do next”, not “what happened inside the server”:

  • DUPLICATE_SUBMIT is separate from ACCEPTED. Both have HTTP status 202, but when the frontend receives DUPLICATE_SUBMIT it keeps the form locked instead of treating it as a newly accepted request.
  • “Awaiting manual handling” is split into two hints depending on whether it can recover on its own. If the request stopped at a step that can be retried on the original MR, the hint is “you can keep waiting”; if it stopped at a deterministic external failure (such as a pipeline failure), the hint is “contact the maintainers; waiting will not help”. Which steps belong to the latter is defined centrally in a constant set.
  • Read-only preview endpoints return empty data on failure, so they never block the user from continuing to submit.

3.6 Idempotency and long-running operations

Idempotency. Write endpoints use the client-supplied operationId as the idempotency key, relying on a database unique constraint to “look up, insert, and re-read on conflict” (see Section 3.4 of the database post). Among concurrent duplicate requests, only one proceeds; the others return the winner’s result.

Long-running operations. A long-running operation returns HTTP 202 after synchronous validation and then executes in a bounded thread pool. When the pool rejects a task, the failure state is written to the database (see Section 3.2 of the MR lifecycle post). The client can always see an explicit state by polling.

3.7 Observability never interferes with business logic

We set a few rules for metrics code and enforced each one in the implementation:

  • The metrics interceptor’s preHandle always returns true and never writes a response.
  • Metrics code only reads values that have already been resolved. For example, it reads request parameters with request.getParameter rather than reading the request body itself, which would consume the input stream and break later argument binding.
  • Metrics use their own identity resolution. When the identity is not trustworthy (for example, only a bare ID that might be an employee ID is available), the count is simply skipped rather than guessed, and the resolved value never flows back into business logic.
  • Counter writes are wrapped in try/catch, and exceptions are only logged. Tag values are normalized and truncated to 128 characters; empty or invalid dimensions cause the emission to be skipped.
  • Choose when to count by semantics. “Request received” is counted when the request arrives; “successful access” is counted in afterCompletion, and only for 2xx responses.
  • Business outcome metrics are emitted only after the transaction commits, and only when the state actually flipped, avoiding over-counting due to rollbacks or concurrency (see Section 3.7 of the MR lifecycle post).
  • Metrics code sits outside the failure-handling logic. For example, the duration is recorded only after a batch hand-off completes, so even if the metrics code throws, it cannot trigger the batch’s failure settlement.
public void incrementUserCounter(String name, Map<String, String> tags) {
    try {
        Map<String, String> safe = normalizeAndTruncate(tags, 128);
        if (safe == null) { log.warn("skip metric {}: invalid tags", name); return; }
        registry.counter(name, safe).inc();
    } catch (RuntimeException e) {
        log.error("metric {} failed; business request continues", name, e);   // A metrics failure does not affect the business request
    }
}

3.8 Static asset fallback during frontend releases

After an SPA release, already-open old pages request hashed assets that were removed in the new version. Without handling, these requests fall through to the global exception handler and return 500. We address this on both the server and the client:

Server side:

  • index.html is served with Cache-Control: no-store, so the latest asset manifest is always fetched;
  • Hashed assets are served with max-age=365d, public, immutable;
  • When an asset does not exist, the response depends on the request type: page navigation requests (Accept: text/html with no file extension in the path) get a fallback page, while static asset or API requests get a 404 JSON. Both responses set no-store.

Client side:

  • Listen for script and stylesheet load errors in the capture phase; when a core asset fails to load, reload the page with a cache-busting parameter;
  • Record the time of the last automatic reload in sessionStorage. Within 30 seconds, do not reload automatically again; show a “Refresh page” button instead, to avoid a reload loop.

We also wrote smoke scripts for these response headers and status codes and run them in the pre-release environment.

4. Results

  • The fallback behavior of every cross-cutting concern is explicitly defined, and documented in code comments and the configuration guide. When configuration is cleared or wrong, rate limiting degrades to no limiting, access switches degrade to closed, and authorization degrades to deny.
  • Rate limiting and authorization complete before argument binding, without changing the response shape of any existing endpoint.
  • Rate-limited requests do not pollute business metrics; they are counted by a separate metric.
  • Exceptions in metrics code do not affect business requests. Metrics code does not read the request body, does not write responses, and does not take part in business branching.
  • Error responses are distinguished by “next action”. Users can tell “duplicate submission”, “you can keep waiting” and “contact the team” apart, while internal details appear only in server logs.
  • During frontend releases, old pages recover on their own, and 500s caused by missing assets no longer occur.

5. Known Limitations and Next Steps

  • Rate limiting is per instance. The effective global quota is roughly “per-instance threshold × number of instances”. In addition, when falling back to address-based limiting, we take the first hop of X-Forwarded-For, which clients can forge. We plan to use the hop appended by a trusted proxy instead, or to rate-limit uniformly at the gateway.
  • The identity resolution logic exists in three copies with subtle differences. One of them does not truncate email-style accounts, which can give the same person two tag values in metrics. We plan to consolidate them into one shared resolver.
  • The mapping from exceptions to HTTP status codes is inconsistent. Some business exceptions return 500; one authorization denial goes through a generic service exception and returns 500 instead of 403; and the fallback handler returns exception messages to the client verbatim. We plan to unify the error model so that the fallback handler returns only a generic message.
  • Some external SDK calls have no timeout and no circuit breaker. When an external dependency slows down, request threads are held for a long time. We plan to set explicit timeouts on all external calls and add isolation for critical dependencies.
  • Idempotency records only the operationId and does not replay the original response. If the client does not supply an operationId, the server-generated ID contains a timestamp, so retried requests cannot be deduplicated.
  • The API metrics interceptor itself has no try/catch. Whether it truly cannot affect business logic depends on the tracing and metrics libraries not throwing. In addition, some user endpoints are outside the coverage of the API metrics.