Freetrade Senior Software Engineer Luke Smith talks about the nuts and bolts of our brokerage platform
Invest by Freetrade is our proprietary investment platform, which we built from the ground up. Many requirements needed to be met to ensure we have accurate data and that the platform could scale to meet the growth of the company.
The platform also looks after your assets. This means it's the most important system we run and we have to make sure that every movement of cash and stock is accounted for.
We understood early on that having access to a stream of information about each account and order would provide us with the flexibility and insights needed to build a brokerage platform for the modern age.
Traditionally a relational database might be used, typically an SQL database, storing data normalised across multiple tables. The individual records are created, updated, or deleted, as necessary (known as CRUD operations). These records store only the current state of your data, as changes occur the history of it is lost. Looking at a record in this structure it is impossible to understand how, or more importantly why, the data reached this current state.
Using a basic table structure we can see that someone's account has a balance of 10, and this was updated on 1st January 2020. We can’t deduce any other information from this.
Event sourcing is different. Each change to your model is captured as an explicit event within what’s known as an aggregate. These events contain the data that is a side-effect of the operation, but importantly the name of each event gives an indication of why or what caused that change. Historical events are never mutated, with new events appended to the underlying store.
By storing explicit events we can see that the same account above had deposits, withdrawals, and trades made against it.
DepositedFunds £100
BoughtStock AAPL £30
WithdrewCash £60
We end up with a balance of £10 as the current state. The current state does not need to be stored (we’ll discuss this more later), giving us a single source of truth about what happened to the account.
A benefit of this is we can also source the events to any historical point in time, meaning we can see what an account looked like at 4:22pm on Thursday 6th June 2019. This gives us more power and flexibility in investigating issues, but also helps us generate the compliance reports and statements a financial company must produce.
The number of events within a specific aggregate can grow over time. For orders, the lifetime is relatively short lived with a handful of events. But an account is a long life aggregate, which can grow to contain many thousands of events relatively quickly.
10s of millions of events overall on customer accounts
10s of thousands on our busiest individual customer accounts
In order to scale and handle this we snapshot what an aggregate looks like on a periodic basis. This snapshot contains only the information necessary to make business decisions within the aggregate. When we need to load an aggregate we first load the snapshot, and then source any unseen events from the event store.
What's in an event
The anatomy of an event contains a lot of information about the outcome of an action. These are the business facts as a result of making a change.
Anything that was known, or calculated by the operation, at that point must be recorded on the event. If it’s not, you can assume it’s lost. This allows the event, and the information it provides, to be the truth for downstream processes to trust. For example, the stamp duty owed as a result of the order is recorded as a fact on an event.
Because Invest by Freetrade is an accounting platform at heart, it’s important that any transaction involving an asset is recorded following double-entry accounting principles. The events act as the method to record the transactions.
An event crediting an account with £100 would also contain information about the other party in the transaction (where the money is being debited from). Likewise the event recording the debit will contain details about the account being credited. Both tied together with a shared transaction reference.
When transactions occur between aggregates we don’t write the events to both at the same time. This would introduce contention in the platform. A customer account has relatively little throughput, but many of the system accounts are often involved in operations where in doing so would introduce a bottleneck that would leak its way back into the UX. As a result we delay when the transaction is recorded via a queue. This introduces the concept of eventual consistency.
We can verify the system is correct by replaying the events across the various accounts, and verifying that the system has eventually recorded both sides of the transaction. The sum of the credit + debit equals zero.
Hopefully this has given you an understanding of Invest by Freetrade, and some of the design decisions we made. Future posts will go into more detail around the challenges and solutions we’ve implemented around using event sourcing in a serverless architecture.
Thank you! Please check for your confirmation email.
Oops! Something went wrong while submitting the form.
Join the 25,000+ investors getting our take on the markets
Almost there! Please check your inbox to confirm subscription
Freetrade Senior Software Engineer Luke Smith talks about the nuts and bolts of our brokerage platform
Invest by Freetrade is our proprietary investment platform, which we built from the ground up. Many requirements needed to be met to ensure we have accurate data and that the platform could scale to meet the growth of the company.
The platform also looks after your assets. This means it's the most important system we run and we have to make sure that every movement of cash and stock is accounted for.
We understood early on that having access to a stream of information about each account and order would provide us with the flexibility and insights needed to build a brokerage platform for the modern age.
Traditionally a relational database might be used, typically an SQL database, storing data normalised across multiple tables. The individual records are created, updated, or deleted, as necessary (known as CRUD operations). These records store only the current state of your data, as changes occur the history of it is lost. Looking at a record in this structure it is impossible to understand how, or more importantly why, the data reached this current state.
Using a basic table structure we can see that someone's account has a balance of 10, and this was updated on 1st January 2020. We can’t deduce any other information from this.
Event sourcing is different. Each change to your model is captured as an explicit event within what’s known as an aggregate. These events contain the data that is a side-effect of the operation, but importantly the name of each event gives an indication of why or what caused that change. Historical events are never mutated, with new events appended to the underlying store.
By storing explicit events we can see that the same account above had deposits, withdrawals, and trades made against it.
We end up with a balance of £10 as the current state. The current state does not need to be stored (we’ll discuss this more later), giving us a single source of truth about what happened to the account.
A benefit of this is we can also source the events to any historical point in time, meaning we can see what an account looked like at 4:22pm on Thursday 6th June 2019. This gives us more power and flexibility in investigating issues, but also helps us generate the compliance reports and statements a financial company must produce.
The number of events within a specific aggregate can grow over time. For orders, the lifetime is relatively short lived with a handful of events. But an account is a long life aggregate, which can grow to contain many thousands of events relatively quickly.
In order to scale and handle this we snapshot what an aggregate looks like on a periodic basis. This snapshot contains only the information necessary to make business decisions within the aggregate. When we need to load an aggregate we first load the snapshot, and then source any unseen events from the event store.
What's in an event
The anatomy of an event contains a lot of information about the outcome of an action. These are the business facts as a result of making a change.
Anything that was known, or calculated by the operation, at that point must be recorded on the event. If it’s not, you can assume it’s lost. This allows the event, and the information it provides, to be the truth for downstream processes to trust. For example, the stamp duty owed as a result of the order is recorded as a fact on an event.
Because Invest by Freetrade is an accounting platform at heart, it’s important that any transaction involving an asset is recorded following double-entry accounting principles. The events act as the method to record the transactions.
An event crediting an account with £100 would also contain information about the other party in the transaction (where the money is being debited from). Likewise the event recording the debit will contain details about the account being credited. Both tied together with a shared transaction reference.
When transactions occur between aggregates we don’t write the events to both at the same time. This would introduce contention in the platform. A customer account has relatively little throughput, but many of the system accounts are often involved in operations where in doing so would introduce a bottleneck that would leak its way back into the UX. As a result we delay when the transaction is recorded via a queue. This introduces the concept of eventual consistency.
We can verify the system is correct by replaying the events across the various accounts, and verifying that the system has eventually recorded both sides of the transaction. The sum of the credit + debit equals zero.
Hopefully this has given you an understanding of Invest by Freetrade, and some of the design decisions we made. Future posts will go into more detail around the challenges and solutions we’ve implemented around using event sourcing in a serverless architecture.