Overview

How might we design a web application utilizing machine learning models to empower Citibank compliance officers to comply with regulations by effectively detecting, investigating, and prosecuting collusive behavior by Forex traders?

Problem

In 2014, several major banks, including Citibank, were fined billions of dollars for failing to adequately monitor their foreign exchange (forex) traders, who were found to be manipulating benchmark exchange rates. The scandal involved traders using private chat rooms to coordinate their actions to manipulate the WM/Reuters Closing Spot Rates. This manipulation allowed them to benefit their trading positions at the expense of the market's integrity

Citibank, along with JPMorgan, HSBC, Barclays, and the Royal Bank of Scotland, faced collective penalties exceeding $1.4 billion imposed by the Commodity Futures Trading Commission (CFTC). The U.S. Department of Justice also took action, resulting in parent-level guilty pleas from several banks, including Citibank, which agreed to a fine of $925 million for its role in the conspiracy to manipulate forex rates​

Training application

Prior to me joining the team, a developer had built this functional proof of concept. Compliance officers would label chat data to train our machine learning models to detect collusive trading behavior. The interface included chat records with timestamps, chat IDs, and content, annotated to highlight interactions from different companies. This tool enabled compliance officers to enhance the accuracy of machine learning models by directly contributing to the identification of suspicious activities.

Building empathy

Throughout the initial efforts, stakeholders actively prevented my direct engagement with internal compliance officers, creating a barrier that led to significant opacity in understanding the true requirements. This lack of direct communication introduced a bias that skewed the project’s direction, favoring the desires of the business stakeholders over the genuine needs of the end-users.

As a result, the project's development was influenced more by business priorities rather than addressing the practical and critical requirements of the compliance officers who would be using the system. This disconnect underscored the challenges of developing a user-centric solution in an environment where key user insights were filtered through intermediaries.

Guerilla user research

Day in the life of a forex trader

Since I wasn’t able to speak directly with compliance officers I watched videos like this one to build up an understanding of what those compliance officers were looking for.

Dumpster diving on Wall Street

I spent time digging through the existing system Citibank had been using to understand and identify the types of behavior we’d need to support.

Building "something"

Kickoff

Around 12 people participated in building the initial map. Utilizing the large whiteboard space (8' tall x 12' wide), participants were briefed on their roles and asked to write stories. Prior workshops were held to assist those with limited experience in story writing. Our goal was to ensure everyone's perspective was considered and heard.

Tasks were organized into categories such as high-level system goals, model building, implementing unique identifiers, microservices, UI/UX design, and review objectives. Key tasks included developing training models for collusion and small talk, creating microservices, and building user interface components.

Sprint board

Key user stories include analysts labeling contextual information within alerts, adjusting behavior preferences in the app, escalating alerts for policy violations, and viewing the history of closed communications. There are also tasks focused on improving system efficiency, such as implementing search functions for alerts, enabling keyboard shortcuts, and exporting alert content for expert review.

Tracking progress

I leveraged the Labels feature in Trello along with the robust capabilities of the third-party app Screenful.me to dynamically build and analyze cumulative flow charts by epic. This approach allowed me to gain a comprehensive overview of task distribution and progress over time.

Release candidate

Borrowing stylistically from Inbox by Google, we designed our interface to provide a high level of insight and usability for compliance officers. The Alerts panel on the left categorizes and lists reviewed alerts, each with a summary of the conversation and classification for easy scanning. The Communication panel on the right provides detailed views of specific conversations, showing participants, timestamps, and highlighted text for quick identification of key statements.

Building trust

Given that none of the stakeholders had experience building user-centric software, it took considerable effort to shift their perspective. Over the course of 4 months and 11 meetings totaling 9.5 hours, I finally convinced leadership of the true state of the project. They ultimately relented and allowed me to speak directly with the compliance officers.

Building empathy

I traveled to Buffalo, NY, for three days to speak with Level 1 analysts. These analysts, typically fresh out of college, were often regarded as mere digital ditch diggers, tirelessly sifting through alerts in search of hidden gems. Despite their critical role, they were frequently undervalued and underappreciated, seen as the frontline workers in the battle against non-compliance.

In contrast, Level 2 analysts worked from offices in Jersey City, NJ. These individuals often came from backgrounds as former traders or law enforcement officers, bringing a wealth of experience and a different perspective to the table. Their expertise allowed them to handle more complex and nuanced cases, making them a vital part of the compliance process.

Post-interview output

What we learned

After speaking directly with the people involved, the amount of information gathered was frankly overwhelming. As I'll explain below, a significant portion of what we were told either wasn't true, or there were events happening that we were unaware of prior to the interviews.

Level 1 analyst

A day in the life

Reviewing an alert

Level 2 analyst

A day in the life

How each level works together

Design

Whiteboard wireframe

Three column layout

Feature Breakdown

Conversation participant card

Ancillary information about the actual participants often informed the users whether the identified malfeasance was actually bad or not. The model would identify something that, under normal circumstances, would be a violation.

Audit trail

The activity log provides a comprehensive overview of actions taken by analysts over a two-day period. It meticulously tracks various stages such as classification, viewing, rejection, confirmation, and referral of alerts. Each action is timestamped and accompanied by detailed notes, offering a clear record of the process. To enhance readability and ensure quick comprehension, icons are used to represent different actions, creating a visual and scannable format.

Investigate mode

This view is particularly useful for compliance officers because it offers a comprehensive overview of the employee’s connections and communication patterns. By including eComms that didn’t trigger alerts, officers gain a full picture of the employee’s interactions, helping to identify potential risks or unusual behavior that may not be immediately evident from alerts alone. The organized and accessible format ensures that all relevant information is easy to navigate, allowing compliance officers to efficiently monitor and investigate compliance issues.

Labeling data

The highlighter menu enables analysts to classify specific behaviors by selecting from options such as "Possible Collusion," "Possible Gifts & Entertainment," "Possible Manipulation," and "Possible Misconduct." This granular labeling system helps in accurately identifying and categorizing various types of inappropriate behavior, thereby improving the training and effectiveness of compliance models.

Outcome

CItibank was ultimately released from having to pay any of the additional $2 billon fine and the SEC was satisfied with the protections they'd put in place. This application is in use by 40,000 persons all around the globe.