Bank Statement Parsing

Plane task: MHPAY-3

This document describes the bank statement parsing architecture in MyHoreca Pay.

It defines how uploaded Indonesian bank statement files are detected, parsed, normalized, validated, reported to operators, and extended with new bank adapters.

Related documents:

• bank-import-pipeline-review.md — current pipeline review for MHPAY-1.
• normalized-bank-transaction-model.md — normalized transaction model for MHPAY-2.
• bank-adapter-development-checklist.md — practical runbook and mandatory tests for new adapters (MHPAY-27).

Purpose

The bank statement parsing subsystem converts bank-specific files into a shared internal transaction model.

The system must support many Indonesian bank formats without breaking the shared import pipeline.

Core principles:

• each bank/file format is implemented as an adapter;
• adapters detect, parse, and normalize their own format;
• parsing must be explainable to operators;
• raw source data must remain traceable;
• invalid rows must produce clear warnings/errors;
• the pipeline must remain independent from Syrve/iiko payment posting logic;
• new bank support must be added without changing existing adapters unless necessary.

High-level flow

Operator uploads bank file
  -> API receives file stream
  -> import service creates/imports context
  -> format detector evaluates registered adapters
  -> best adapter is selected
  -> parser executes selected adapter
  -> adapter returns normalized transactions + warnings/errors
  -> import service persists import, transactions, detection and quality counters
  -> API returns import result
  -> UI shows import quality to operator
  -> downstream matching/allocation can start

Main backend components

Component	Responsibility
IBankAdapter	Bank/format-specific detection and parsing contract.
BankAdapterDescriptor	Adapter metadata: bank code, display name, format code, AdapterVersion / ParserVersion / NormalizerVersion, status, capabilities, notes.
IBankStatementFormatDetector	Selects the most likely adapter for an uploaded file.
IBankStatementParser	Runs parsing through the selected adapter.
IBankStatementImportService	Orchestrates upload/import/persist operation.
IBankStatementSupportedBanksService	Exposes supported banks and formats to API/UI.
IBankStatementQualityService	Provides import quality, balance checks, and duplicate candidate workflow.
NormalizedBankTransaction	Canonical transaction model produced by adapters.

Adapter contract

Every bank format must be implemented through IBankAdapter.

An adapter must:

• expose a stable descriptor;
• declare bank code and format code;
• declare AdapterVersion, ParserVersion, and NormalizerVersion (see bank-adapter-changelog.md);
• score detection confidence;
• decide if it can parse the file;
• parse the source stream;
• return normalized transactions;
• return warnings and errors;
• return parse confidence;
• preserve raw source fields for audit.

Adapters must not:

• call Syrve/iiko APIs;
• create payment allocations;
• decide which Syrve document is paid;
• silently drop invalid rows;
• hide raw source values;
• invent counterparties when the source is ambiguous.

Adapter descriptor

Each adapter descriptor should provide enough metadata for API, UI and support diagnostics.

Recommended descriptor fields:

• bank code, for example DEMO, PERMATA, BCA;
• bank display name;
• format code, for example CSV, TXT, MT940;
• adapter name;
• AdapterVersion, ParserVersion, NormalizerVersion (semver strings);
• status: experimental, beta, supported, deprecated;
• supported file extensions;
• supported content types;
• capabilities;
• limitations/notes.

The UI should be able to show operators which bank formats are supported and what limitations exist.

Format detection

Format detection chooses the most appropriate adapter before parsing.

Detection should use stable signals, for example:

• file extension;
• content type;
• header row;
• mandatory columns;
• bank-specific markers;
• account/statement markers;
• MT940 tags;
• date/amount/reference layout;
• sample row structure.

Detection result should include:

• selected bank code;
• selected format code;
• adapter name;
• adapter/parser/normalizer versions from descriptor;
• detection confidence;
• detection notes or warnings when useful.

Confidence guidance:

Confidence	Meaning
0.90 - 1.00	Strong match. Bank/format-specific markers are present.
0.70 - 0.89	Probable match. Most required signals are present.
0.40 - 0.69	Weak match. Manual review or fallback may be needed.
< 0.40	Should not be selected automatically.

If multiple adapters score similarly, the detector should prefer the more specific adapter and record ambiguity.

Parsing

Parsing converts the selected bank file into normalized transactions.

The parser must:

• read the source safely;
• handle expected encoding/date/number formats;
• validate required fields;
• map source fields to NormalizedBankTransaction;
• collect row-level warnings and errors;
• calculate parse confidence;
• preserve raw source values.

Parsing should be deterministic. The same input file should produce the same normalized result.

Normalized transaction model

All adapters must output NormalizedBankTransaction records.

Required fields:

• Id;
• ImportId;
• SourceBank;
• SourceFormat;
• Currency;
• TransactionDate;
• Amount;
• Direction;
• RawDescription;
• RawFields.

Important optional fields:

• AccountNumber;
• AccountName;
• ValueDate;
• CounterpartyRawName;
• BankReference;
• CustomerReference;
• PaymentPurpose;
• BalanceAfter.

Detailed field rules are documented in normalized-bank-transaction-model.md.

Amount and direction normalization

Adapters must normalize amount and direction consistently.

Rules:

• Amount is positive.
• Direction = Credit means money entered the bank account.
• Direction = Debit means money left the bank account.
• Negative source amounts must be converted into positive amount plus correct direction.
• Separate debit/credit columns must not both be populated in one row.
• If direction cannot be determined, the row must fail with a clear error.

Date normalization

Each adapter must explicitly define supported date formats.

For Indonesian bank exports, expected date forms may include:

• dd/MM/yyyy;
• dd-MM-yyyy;
• yyyy-MM-dd;
• dd MMM yyyy;
• bank-specific localized month names.

Adapters should avoid guessing. Strict parsing with clear row errors is safer than silently producing incorrect dates.

Raw field preservation

Every parsed transaction must preserve source traceability through RawFields.

Recommended values:

• source line number or row number;
• raw line when practical;
• original source column names and values;
• relevant parser assumptions;
• source tag/segment information for text formats.

This is required for:

• support diagnostics;
• operator explanations;
• audit trail;
• duplicate investigation;
• future adapter fixes.

Warnings and errors

Warnings are non-blocking issues. Errors are row-level or import-level failures.

Examples of warnings:

• missing counterparty;
• missing optional reference;
• assumed currency;
• missing value date;
• missing running balance;
• description looks too short;
• duplicate candidate detected.

Examples of errors:

• invalid date;
• invalid amount;
• unknown direction;
• missing mandatory columns;
• broken row structure;
• both debit and credit populated;
• unsupported file format.

Operator messages must be practical. The user should understand which row failed and why.

Quality report

The quality report should summarize import health.

Expected information:

• import ID;
• bank code;
• format code;
• adapter name;
• detection confidence;
• parse confidence;
• total rows;
• parsed transactions;
• skipped rows;
• warnings count;
• errors count;
• duplicate candidate count;
• opening balance when available;
• closing balance when available;
• calculated closing balance;
• balance match result;
• warning details;
• error details;
• metadata.

The quality report is the main operator-facing explanation layer for parsing.

Duplicate detection

Duplicate detection prevents repeated imports from creating duplicate payment evidence.

Useful duplicate signals:

• source bank;
• account number;
• transaction date;
• amount;
• direction;
• currency;
• bank reference;
• customer reference;
• normalized description;
• normalized counterparty name.

The system should support manual confirm/dismiss duplicate candidate workflow.

Supported banks registry

The supported banks API/UI should be backed by registered adapters.

For each adapter, expose:

• bank code;
• bank display name;
• format code;
• adapter name;
• status;
• supported extensions;
• notes/capabilities.

This allows operators to know before upload whether their bank file is supported.

Adding a new bank adapter

Use the full Bank Adapter Development Checklist (MHPAY-27) as the authoritative runbook. It covers descriptor fields, detection scoring, date/amount/direction rules, raw fields, warnings/errors, sanitized fixtures, mandatory tests, registry/DI, and verification commands.

Starter template (not compiled, not registered): backend/src/MyHoreca.Pay.BankStatements/Adapters/_TemplateBankAdapter.example.cs.

Short summary:

1. Collect real sample files (local only — never commit).
2. Sanitize sensitive data → fixtures under Fixtures/BankStatements.
3. Copy template → implement IBankAdapter with explicit detection and parsing.
4. Register in BankAdapterRegistry and DI.
5. Add mandatory unit tests (see checklist section 12).
6. Update supported banks API/docs.
7. Run dotnet test --filter BankStatement and docs build.

Test requirements

Each real adapter should have tests for:

• positive detection;
• negative detection;
• successful parsing;
• invalid date;
• invalid amount;
• unknown direction;
• missing mandatory column;
• missing optional counterparty/reference warnings;
• raw fields preservation;
• parse confidence calculation.

Test fixtures must be sanitized and must not contain real customer secrets, passwords, private bank data, or production credentials.

Current adapter baseline

Production adapters (registered in DI / BankAdapterRegistry):

• CorporateExplicitDebitCreditCsvAdapter (CIMB)
• MultiAccountCrDbCsvAdapter (BCA)
• PermataCsvAdapter / PermataTextAdapter / PermataMt940LikeAdapter
• MandiriAccountStatementCsvAdapter

New adapters should follow _TemplateBankAdapter.example.cs and the development checklist — not invent a synthetic catch-all format.

Phase 1 implementation order

Recommended order:

1. Keep MHPAY-1 and MHPAY-2 documents as baseline.
2. Use this document as architecture rulebook.
3. Choose first real Indonesian bank sample.
4. Implement first real CSV adapter.
5. Add tests and fixtures.
6. Expose supported bank format status.
7. Add quality report improvements.
8. Add duplicate detection improvements.

Open gaps

Current gaps for Phase 1:

• no full real Indonesian bank adapter coverage yet;
• demo CSV parsing is too simple for real quoted CSV/locale cases;
• bank-specific date/amount conventions need to be documented per adapter;
• supported banks UI/API should clearly show adapter status;
• operator quality report should become more visible in UI;
• duplicate detection needs real-world validation with repeated imports.

Acceptance criteria

MHPAY-3 is complete when:

• architecture document exists at docs/docs/architecture/bank-statement-parsing.md;
• it explains adapter-based parsing;
• it documents detection, parsing, normalization and quality reporting;
• it explains how to add a new bank adapter;
• it defines test expectations;
• it links conceptually to MHPAY-1, MHPAY-2, MHPAY-4, and MHPAY-5.

PDF and photograph recognition branch

The existing import lifecycle branches only after staged streaming storage and SHA-256 duplicate detection. CSV/TXT/MT940 keep using existing adapters. PDF/JPEG/PNG/WEBP create one PostgreSQL-backed recognition run and never enter a structured statement adapter.

The leased worker resolves the single global Mistral model by input kind, opens the local stream once and uploads one temporary file to /v1/files with purpose ocr. A single /v1/ocr request returns document classification and extracted operations through strict document annotation plus OCR pages/blocks. The generated Mistral upload method is not used because its byte[] contract would buffer the complete document; the provider project uses IHttpClientFactory, multipart StreamContent and typed Newtonsoft DTOs. Base64 and dynamic JSON are prohibited.

Rows are drafts: RecognizedValuesJson is immutable model output and CurrentValuesJson is the operator draft. Approximate normalized boxes link rows/evidence to pages; a null box is valid. Only atomic confirmation creates BankTransaction, duplicate candidates, matching and quality data. Pending/processing/review originals are excluded from cleanup; a missing original returns recognition_original_file_unavailable without deleting audit/correction/learning metadata.