> ## Documentation Index
> Fetch the complete documentation index at: https://docs.drpn.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Reconciliation engine

> How Darpan turns source data into generated reconciliation output.

This is where a run actually spends its time, so the two passes — base diff and rule evaluation — are worth understanding before making engine changes.

The reconciliation engine is the run pipeline. It stages source data, normalizes records, compares matching keys, applies rules, and writes generated output for review.

## High-level flow

```text theme={null}
UI workflow
  -> JSON-RPC facade service
  -> access and tenant checks
  -> saved run, RuleSet, mapping, or source records
  -> source data staging
  -> Spark dataset preparation
  -> base diff and optional rule execution
  -> generated output descriptor and file
  -> UI result view
```

## Run responsibilities

| Stage            | Responsibility                                                                  |
| ---------------- | ------------------------------------------------------------------------------- |
| Setup and access | Persist run setup and resolve user, company, and tenant scope.                  |
| Source staging   | Save uploaded files, text payloads, or configured-source inputs for processing. |
| Base comparison  | Normalize records and compute missing-object or matched-pair output.            |
| Rule evaluation  | Apply RuleSet rules to compare-ready data and matched pairs.                    |
| Output storage   | Store generated output metadata and files for review.                           |

## Current execution paths

Darpan has two execution paths.

| Path         | Role                                                                                   |
| ------------ | -------------------------------------------------------------------------------------- |
| RuleSet path | A saved run points to a `RuleSetCompareScope` with compare sources and optional rules. |
| Mapping path | A reconciliation mapping describes source members and comparison behavior.             |

Darpan supports both paths. New run contracts center on `RuleSetCompareScope`; existing services and release history may still reference mappings.

## RuleSet compare flow

The RuleSet path is built from these records:

* `RuleSet`
* `RuleSetCompareScope`
* `RuleSetCompareSource`
* `Rule`
* `ReconciliationRun`
* Generated output descriptor records

The compare flow is:

1. The facade service creates or loads a saved run and its compare scope.
2. Generic reconciliation saves uploaded or text payloads in a tenant-scoped temp location.
3. `prepare#RuleSetCompareScope` resolves compare sources and builds normalized Spark datasets.
4. `reconcile#RuleSetCompareScopeBaseDiff` finds base missing-object differences and matched pairs.
5. `reconcile#RuleSetCompareScope` preserves base diffs and appends rule-generated diffs from matched pairs.
6. The backend writes result data and summary metadata as a generated output.

## File and source handling

The backend can compare uploaded files, text payloads, and configured sources. CSV and JSON inputs are normalized before comparison. The comparison layer receives structured data and saved run context, not UI-specific state.

Generic file reconciliation is routed through:

```text theme={null}
reconciliation.ReconciliationGenericServices.reconcile#GenericFiles
```

RuleSet comparison uses:

```text theme={null}
reconciliation.ReconciliationCoreServices.prepare#RuleSetCompareScope
reconciliation.ReconciliationCoreServices.reconcile#RuleSetCompareScope
```

The legacy mapping path still routes through:

```text theme={null}
reconciliation.ReconciliationCoreServices.reconcile#FilesByMapping
```

## Generated outputs

Generated outputs are not only downloaded files. The backend stores descriptor metadata so the UI can list, open, and delete outputs consistently.

Generated-output metadata can include:

* Saved run ID and display name
* RuleSet and compare-scope IDs
* Source labels
* Output location
* Diff counts and summary fields
* Created user and tenant scope

Generated output operations are exposed through facade services such as:

```text theme={null}
facade.ReconciliationFacadeServices.list#GeneratedOutputs
facade.ReconciliationFacadeServices.get#GeneratedOutput
facade.ReconciliationFacadeServices.delete#GeneratedOutput
```

## Run-completion notifications

Runs that reach a terminal state alert the owning tenant so sync issues surface without anyone watching the run history page. Destinations are resolved per run rather than per tenant, which is what lets one person follow a single run without changing shared configuration.

1. A run reaches a terminal state through one of three paths: a manual saved-run diff (`facade.ReconciliationFacadeServices.run#SavedRunDiff`), a scheduled automation execution, or SFTP polling. Terminal states are `AUT_STAT_SUCCESS`, `AUT_STAT_FAILED`, `AUT_STAT_NO_DATA`, `AUT_STAT_SKIP_DUP`, and `AUT_STAT_CANCELLED`.
2. The completion path calls `darpan.reconciliation.notification.TenantNotificationSupport`, which resolves destinations through `resolveDestinationChatSpaces`: the automation's linked `chatSpaceId`, plus the snapshotted space on every `darpan.reconciliation.ReconciliationRunNotifySubscription` row for the run.
3. That set is deduplicated and filtered to `darpan.reconciliation.TenantChatSpace` rows that are active, hold a webhook URL, and belong to the run's own tenant. A dropped *automation-linked* space is logged as a warning; a dropped subscriber space is expected churn and stays silent.
4. The helper builds the message through `reconciliation.ReconciliationNotificationServices.build#RunCompletedPayload` and posts it to each surviving destination.

Delivery is claim-then-deliver: a single claim per run-result means at most one notification pass even when several stages report completion. Subscription rows are purged once the run is terminal. The payload includes the run name, tenant label, result ID, a link back to the run result, the difference count, and the only-in-source counts. The run-result link base URL comes from the `DARPAN_APP_BASE_URL` environment variable or the `darpan.app.baseUrl` property, falling back to the first allowed origin.

The registry and the per-user default are configured through:

```text theme={null}
facade.SettingsFacadeServices.list#TenantChatSpaces
facade.SettingsFacadeServices.save#TenantChatSpace
facade.SettingsFacadeServices.delete#TenantChatSpace
facade.SettingsFacadeServices.get#UserNotificationDefault
facade.SettingsFacadeServices.save#UserNotificationDefault
facade.ReconciliationFacadeServices.subscribe#RunNotification
facade.ReconciliationFacadeServices.unsubscribe#RunNotification
```

`save#TenantChatSpace` requires tenant write access; `save#UserNotificationDefault` and the subscribe pair are personal preferences and do not. `delete#TenantChatSpace` hard-deletes only unreferenced spaces — a space an automation or subscription points at must be deactivated instead.

The webhook URL must be an `https` Google Chat space-messages endpoint on `chat.googleapis.com` with `key` and `token` query parameters. It is stored encrypted and is write-only: facade responses never return it, masked or otherwise.

The single-webhook `darpan.reconciliation.TenantNotificationSetting` entity this replaced is retired. `reconciliation.ReconciliationNotificationServices.migrate#TenantNotificationSettings` is the one-time v1.2.0 upgrade that copies each tenant's old webhook into a space named `Default space` and links that tenant's space-less automations to it. It is idempotent and internal only (`allow-remote="false"`). The product-side setup flow is described in [Manage tenant settings](/guides/manage-tenant-settings).

## Stuck-run recovery

If a JVM crash or container restart kills a run mid-flight, its run-result row can be left in a running state forever, which blocks re-triggering. An internal scheduled reaper handles this:

```text theme={null}
reconciliation.ReconciliationAutomationServices.sweep#StuckReconciliationRuns
```

It flips `ReconciliationRunResult` and `ReconciliationAutomationExecution` rows still in running or pending state, whose last update is older than a threshold (default 120 minutes), to failed with an explanatory status message. The service is internal only (`allow-remote="false"`) and is not part of the JSON-RPC surface.