Technical release notes Q2 / 2023
Released: July 2023
Technical release notes aim to communicate the restructuring we do for the FA Platform behind the scenes every quarter. The technical release notes focus on the improvements not directly visible on the user interface. Topics covered include performance, security, reliability, quality, and new integrations.
Performance
Overview
FA Solutions is committed to building an enterprise-grade platform and we are placing particular emphasis on improving the performance and scalability of the system in 2023. Our goal is to deliver significant improvements in our 2023 Q2, Q3, and Q4 releases, and to address all impactful performance issues that FA Platform users may encounter. Therefore, the 2023 Q2 technical release notes, as well as the upcoming release notes, will prominently feature the numerous performance improvements that we have been diligently working on.
We have worked with performance-related topics before and will continue to work with them in the years to come. Relative to how we have worked with performance issues in the past, this year we are taking a more deliberate, comprehensive, and proactive approach to them, and focusing additional resources to this important area.
We will not resolve everything during the year, but we expect to make a big difference in this release and the upcoming releases, and we are looking forward to leveling up the performance and scalability of the platform.
Significant speed improvement in FA Back Transactions and Trade orders views
Why?
Searching for transactions and trade orders in FA Back is a frequent process. Depending on the amount of data in the system and the search criteria used, searches may take a long time.
Who is this for?
This improvement is for users who have encountered slow searches when using the Transactions and Trade orders views in FA Back or when using FA Back dashboards that display transaction or trade order listings.
Details
Several significant performance improvements and optimizations have been made to the Transactions and Trade orders views and the listings in Overview and dashboards. As a result, all kinds of transaction and trade order searches should be noticeably faster than before, with certain searches (e.g. tag-based searches with otherwise broad search criteria) performing significantly better.
We achieved this through a combination of different improvements and optimizations. One improvement was to not automatically calculate aggregate figures (e.g. total amount, trade amount et cetera) when searching transactions or trade orders. Fetching this data can have a significant impact on performance. We now default to not fetching the aggregated figures, but we added a checkbox to enable this feature. Other significant improvements include the optimization of tag and portfolio group use in search criteria and technical improvements related to transaction listings.
Significant performance improvement in task lists in FA Back
Why?
Users with a large number of tasks in their system encountered significant performance issues in the Tasks view of FA Back.
Who is this for?
This is especially for users who work with a large number of tasks in FA Back.
Details
Performance issues could occur even when the number of tasks matching task filtering criteria was small because the overall number of tasks in the system was large. We made technical improvements to how data is loaded from the database, making task listings work significantly faster.
Java version upgrade: future-proofing the FA Platform
Why?
Java is our main back-end services programming language. To future-proof the platform and take advantage of features in newer Java versions, we switched to using Java 17 for most of our back-end services.
Who is this for?
This affects all FA Platform users as they rely on services written in Java.
Details
The most immediate impact of the Java version upgrade is that we simultaneously started to use a more modern Java garbage collector (ZGC). The main advantages of ZGC are that it allows us to use our cloud resources more efficiently and helps us to avoid situations where a normally fast operation suddenly and seemingly randomly takes lots of time. Upgrading to Java 17 is also positive for security: Java 17 is a long-term-supported language version that will be supported by vendors for many years to come.
Improved performance for Corporate actions in certain cases
Why?
We found a performance bottleneck in the way we run certain Corporate actions.
Who is this for?
This is for all users of the FA Platform who schedule Corporate actions or run dividend Corporate actions.
Details
The specific bottlenecks we resolved had to do with saving transactions inefficiently to the database and unnecessarily loading currency data while calculating what dividends to generate. We intend to follow up with similar improvements for other types of Corporate actions in a future release.
Reliability and performance improvements in portfolio groups
Why?
Some users encountered issues with the performance and reliability of portfolio groups, especially with groups containing a large number of portfolios.
Who is this for?
This is especially for users who use portfolio groups, particularly those with large groups or groups containing duplicate records.
Details
In this release, we addressed two problems related to portfolio groups:
Sometimes using a portfolio group as a selection, the search or query criteria can be very slow. This was particularly prevalent with portfolio groups containing tens of thousands of portfolios. The issue had to do with the database failing to properly optimize query execution. Some of those queries now work significantly faster, but there is more work to be done in this area.
Sometimes we had a portfolio group containing a portfolio multiple times. This was not intentional and could cause incorrect results when operating with the portfolio group. We added error handling and a database-level restriction to prevent duplicate portfolio group entries.
Slight improvement in FA Back Overview performance
Why?
FA Back Overview is a crucial part of the FA Platform and is used extensively, but loading the necessary data can be slow, especially when viewing large portfolios or portfolio groups.
Who is this for?
This is especially for users who work with large portfolios or portfolio groups.
Details
Several smaller performance improvements and optimizations have been made. These include how Overview collects and shows position data for the top half of the Overview page and how transactions and trade orders are handled in the bottom half of the Overview page. The aim of these improvements and optimizations is to reduce the loading speed of the Overview, but the results are marginal so far. We intend to follow up with further improvements in future FA releases.
Reused data in FA Client Portal views
Why?
We noticed there was an opportunity to fetch data to both the Overview and Holdings views with a single API request.
Who is this for?
This improvement is for all users of the FA Client Portal.
Details
Since version 1.5 of the FA Client Portal, Overview uses data from Analytics in FA Back. This makes it possible to use the same query in both Overview and Holdings views (for Contact and Portfolio, respectively) since they rely on the same underlying data. Overview and Holdings data is now fetched with a single API request, meaning no request is made when moving from Overview to Holdings. The result is a reduced loading speed.
Improved loading time in Trading view in FA Client Portal
Why?
We noticed that navigating into the Trading view in the FA Client Portal was slow when there were many tradable securities.
Who is this for?
This improvement is for all users of the FA Client Portal who allow trading.
Details
Navigating into and out of the Trading view with many tradable securities used to appear slow, negatively affecting user experience. In version 1.5, the loading time is significantly decreased when navigating into the Trading view. Various optimizations have been made. For example, we no longer load the same data twice and no longer render the same row three times.
Automatically log and stop long-running queries
Why?
Sometimes users can cause very long-lasting queries to run against the FA database which can impact the overall performance of the system. We wanted a way to automatically log and stop such queries.
Who is this for?
This improvement is for all users of the FA Platform.
Details
FA users can write their own queries, making queries that run for many hours or even multiple days possible. In extreme cases, environments can have multiple of such queries running in parallel. A query that runs for several hours is unlikely to be useful since most users do not want to wait that long.
We implemented an automatic procedure that logs and stops overly long-running queries. For system administration purposes, a log entry is made whenever queries run for more than five minutes, and queries that have been running for longer than an hour are stopped. This feature is currently an optional package available as “Slow query logger” in the “Processes” section of the AppStore. In the future, we intend to automatically install this feature for all users with a 12-hour maximum query duration.
Other performance fixes
Large transaction and trade order imports are now more efficient. This was achieved by combining multiple inserts/updates to a single database command. This also improves the speed of extensions that use our importer to add transactions or trade orders to the FA Platform.
Updating security prices via imports is now faster, especially in environments with a large number of securities.Updating a market price always involves finding the corresponding security which can be found with various search criteria. Some of the criteria, including the default criteria of the price importer, were slow due to poor indexing. We added indexing for the ISIN code, trade code, and update codes (1-3) of securities. These indexes will also improve search speed when looking for securities based on one of these criteria.
We added parallel processing to a messaging-based feature for launching flowable workflows. Instead of reacting to requests to launch flowable workflows one-by-one, we process up to five requests at a time, increasing the throughput.
In some cases, customised decision tables and other types of business rules can cause performance bottlenecks. We improved our monitoring capabilities for decision tables and rules to more easily spot these bottlenecks.
Reliability
Improved reliability of position calculation
Why?
Position calculation, also known as “report calculation”, is a core feature of the FA Platform. Some users encountered reliability issues with this feature, particularly with the automatic nightly position calculation.
Who is this for?
This improvement is for all users of the FA Platform who have encountered issues with their nightly position calculation.
Details
We improved the reliability of position calculation in two ways:
The way we schedule and handle retries of position calculation requests is now more robust. We also introduced filtering to recognize where we receive overlapping requests to recalculate the same portfolio over and over.
We aim to avoid situations where the backlog of pending calculation requests grows larger than the total number of portfolios in the system. This was possible when the report calculation service received multiple requests to calculate the same portfolio, e.g. as a result of saving transactions to portfolios one by one. The report calculation service now ignores redundant requests.
Security
We implemented a variety of security-related improvements in the FA Platform for the Q2/2023 release. Maintaining the security of the FA Platform is crucial and requires continuous effort. Doing so has been an important part of our development process for a long time.
The Q2/2023 security improvements most notably include our Java version update. We also updated the docker base images that we use to run our Java services. Additionally, several security-related fixes and enhancements were made to specific features and APIs.
Updates to Technical release notes Q2 / 2023
Technical upgrade for 2022-Q2 - 2023-Q1 releases
During 25.-28.7.2023, we applied a Java version update to some of our applications and services for compatibility with upcoming updates to our cloud infrastructure. Specifically, we incremented the Java version in use in our 2022 Q2 - 2023 Q1 releases of FA Back and Report calculation microservice by a single patch version. A similar upgrade was performed for the 2023 Q1 release of FA Front.