Compliance ingestion pipeline

A pipeline that picks up expert-call transcripts from Google Drive, screens them for material non-public information with AI, and keeps an audit trail. I built the AWS infrastructure on my own in Terraform.

TypeScript, Node.js, AWS Lambda, AWS SQS, API Gateway, Terraform, Supabase, Postgres, pgvector, OpenAI

What it does

Channel Dynamics sells market intelligence to public equity investors. Its research is built on expert-call transcripts, and every piece has to be screened for material non-public information (MNPI) before it reaches investors. That screening was manual work for the compliance officer.

The hard part

Transcripts lived in Google Drive, outside the platform, so nothing knew when a new one arrived. The screening had to be explainable for audits, and everything had to fit the team’s TypeScript stack and their existing Postgres database with pgvector.

What I did

  • Set up a Google Drive watch channel that notifies an AWS API Gateway endpoint whenever a transcript is added or changed.
  • Queued each change on SQS, then processed and vectorised the transcripts into Postgres with pgvector.
  • Classified content for MNPI with OpenAI, and stored every result immutably in S3 and Postgres for audit.
  • Built all of the AWS infrastructure myself in Terraform, and owned the full lifecycle from application logic to secure REST APIs.

Result

MNPI classification reached 98% accuracy, and every result can be traced for audit.