Skip to main content
This is where a run actually spends its time, so the two passes — base diff and rule evaluation — are worth understanding before making engine changes. The reconciliation engine is the run pipeline. It stages source data, normalizes records, compares matching keys, applies rules, and writes generated output for review.

​
High-level flow

​
Run responsibilities

​
Current execution paths

Darpan has two execution paths. Darpan supports both paths. New run contracts center on RuleSetCompareScope; existing services and release history may still reference mappings.

​
RuleSet compare flow

The RuleSet path is built from these records:
  • RuleSet
  • RuleSetCompareScope
  • RuleSetCompareSource
  • Rule
  • ReconciliationRun
  • Generated output descriptor records
The compare flow is:
  1. The facade service creates or loads a saved run and its compare scope.
  2. Generic reconciliation saves uploaded or text payloads in a tenant-scoped temp location.
  3. prepare#RuleSetCompareScope resolves compare sources and builds normalized Spark datasets.
  4. reconcile#RuleSetCompareScopeBaseDiff finds base missing-object differences and matched pairs.
  5. reconcile#RuleSetCompareScope preserves base diffs and appends rule-generated diffs from matched pairs.
  6. The backend writes result data and summary metadata as a generated output.

​
File and source handling

The backend can compare uploaded files, text payloads, and configured sources. CSV and JSON inputs are normalized before comparison. The comparison layer receives structured data and saved run context, not UI-specific state. Generic file reconciliation is routed through:
RuleSet comparison uses:
The legacy mapping path still routes through:

​
Generated outputs

Generated outputs are not only downloaded files. The backend stores descriptor metadata so the UI can list, open, and delete outputs consistently. Generated-output metadata can include:
  • Saved run ID and display name
  • RuleSet and compare-scope IDs
  • Source labels
  • Output location
  • Diff counts and summary fields
  • Created user and tenant scope
Generated output operations are exposed through facade services such as:

​
Run-completion notifications

Runs that reach a terminal state alert the owning tenant so sync issues surface without anyone watching the run history page. Destinations are resolved per run rather than per tenant, which is what lets one person follow a single run without changing shared configuration.
  1. A run reaches a terminal state through one of three paths: a manual saved-run diff (facade.ReconciliationFacadeServices.run#SavedRunDiff), a scheduled automation execution, or SFTP polling. Terminal states are AUT_STAT_SUCCESS, AUT_STAT_FAILED, AUT_STAT_NO_DATA, AUT_STAT_SKIP_DUP, and AUT_STAT_CANCELLED.
  2. The completion path calls darpan.reconciliation.notification.TenantNotificationSupport, which resolves destinations through resolveDestinationChatSpaces: the automation’s linked chatSpaceId, plus the snapshotted space on every darpan.reconciliation.ReconciliationRunNotifySubscription row for the run.
  3. That set is deduplicated and filtered to darpan.reconciliation.TenantChatSpace rows that are active, hold a webhook URL, and belong to the run’s own tenant. A dropped automation-linked space is logged as a warning; a dropped subscriber space is expected churn and stays silent.
  4. The helper builds the message through reconciliation.ReconciliationNotificationServices.build#RunCompletedPayload and posts it to each surviving destination.
Delivery is claim-then-deliver: a single claim per run-result means at most one notification pass even when several stages report completion. Subscription rows are purged once the run is terminal. The payload includes the run name, tenant label, result ID, a link back to the run result, the difference count, and the only-in-source counts. The run-result link base URL comes from the DARPAN_APP_BASE_URL environment variable or the darpan.app.baseUrl property, falling back to the first allowed origin. The registry and the per-user default are configured through:
save#TenantChatSpace requires tenant write access; save#UserNotificationDefault and the subscribe pair are personal preferences and do not. delete#TenantChatSpace hard-deletes only unreferenced spaces — a space an automation or subscription points at must be deactivated instead. The webhook URL must be an https Google Chat space-messages endpoint on chat.googleapis.com with key and token query parameters. It is stored encrypted and is write-only: facade responses never return it, masked or otherwise. The single-webhook darpan.reconciliation.TenantNotificationSetting entity this replaced is retired. reconciliation.ReconciliationNotificationServices.migrate#TenantNotificationSettings is the one-time v1.2.0 upgrade that copies each tenant’s old webhook into a space named Default space and links that tenant’s space-less automations to it. It is idempotent and internal only (allow-remote="false"). The product-side setup flow is described in Manage tenant settings.

​
Stuck-run recovery

If a JVM crash or container restart kills a run mid-flight, its run-result row can be left in a running state forever, which blocks re-triggering. An internal scheduled reaper handles this:
It flips ReconciliationRunResult and ReconciliationAutomationExecution rows still in running or pending state, whose last update is older than a threshold (default 120 minutes), to failed with an explanatory status message. The service is internal only (allow-remote="false") and is not part of the JSON-RPC surface.