Sarthak Aggarwal. Say hello

Project 003 · Lending compliance · RAG

RBI rules answered,with the paragraph to prove it.

A question in plain words in. A cited answer from the rule in force out, in about nine seconds, plus which loans break it. The model writes; Python and the database check.

Watch the demo ↓
94%of held-out questions had the right paragraph in the top five
10 / 10questions about replaced rules answered from the rule in force
8 / 8loan-book questions with an exact result from the database
$0.003per question, about nine seconds

v1 · held-out · 40 questions + 10 replaced-rule, 10 off-topic, 20 loan-book · public RBI text, synthetic loan book · not yet hand-checked

Demo · 2:38

The challenge

A compliance team at an NBFC answers to 16 RBI master directions, about 130,000 words, and the circulars that changed them. Is this rule still in force? Which paragraph says so? Which of our loans does it affect?

Today that means searching PDFs, asking a colleague and writing a query by hand. A wrong answer usually means quoting a replaced circular, or a number nobody can trace.

A question-answering service over the RBI rules for NBFCs, from fetching the pages to a cited answer and a loan-book check.

How it works

  1. 01

    Find

    Keyword and meaning search over 1,493 clauses from 22 documents.

    Replaced circulars set aside, then a small model reranks.

    BM25 · OpenAI embeddings · Python

  2. 02

    Answer

    Every sentence must cite a clause.

    Sentences with numbers missing from their source are dropped; off-topic is refused.

    OpenAI · Python

  3. 03

    Check the loans

    The rule is found first, then one SELECT is written from it.

    Checked by a gate, run as a read-only user; the database counts.

    Postgres · sqlglot

  4. 04

    Show

    Answers stream in, with each cited clause beside them.

    Loans, the query in plain words, and an Excel export.

    React · Node · Express

How a request flows

Animated flow diagram of RegPulse

Architecture

Screens

A cited answer, sources beside it
A cited answer, sources beside it
Impact check: loans that break a rule
Impact check: loans that break a rule
Every direction, clause by clause
Every direction, clause by clause
The eval and ablation, inside the app
The eval and ablation, inside the app

What I learned

  1. Measure each part before keeping it. The local reranker looked like a free win; a small model ranked better (MRR 0.93 vs 0.83). The first run also found two bugs, so a fresh test set was written after the fixes.
  2. Real RBI pages are messier than a demo corpus. The PDFs sit behind a bot check, so I parse the HTML. Numbering restarts inside annexes, so 55 paragraphs needed their own IDs.
  3. Still open: answers are judged correct 78% of the time while the right paragraph is found 94% of the time, so the next gains are in the answer step.

Stack

ReactNodeExpressPythonFastAPIPostgresBM25OpenAILangfuseDocker