Why Basic Mathematics and Statistics Still Hold the Key to Unlocking Real Insights in the Era of Quantum Computing

Many modern academic studies concentrate on machine learning, driven by Google's exciting advancements in quantum computing, which bring significant promise. However, a number of these studies overlook traditional mathematical principles in their approach to machine learning. As a result, their findings can be confusing and, in some cases, disappointingly unimpressive.

My latest paper explores a particular example within the field of optimization. Optimization originated in the 1920s as a byproduct of industrial manufacturing, characterized by structured processes like regimented conveyor systems and precisely timed worker inputs at facilities such as Ford's. The field gained momentum in the 1950s, fueled by military research into precision and feedback loops for control systems, and influential work in portfolio management by Markowitz in 1952. The 1980s saw optimization focus on supply chain management, and it expanded into financial markets during the 1990s and 2000s. Today, optimization is most prominently applied in human-computer interaction, striving to guide users toward clicking the enticing "pay now" button through a variety of digital strategies.

As computing costs have decreased, optimization has become increasingly technologically advanced. However, this focus on computational methods often overlooks basic efficiencies that can be achieved through statistics. My latest paper, "Conditional Optimization," aims to demonstrate this point.

The core of the paper revolves around a fascinating application in the field of optimization, known as the "Predict-then-Optimize" problem. This problem addresses the inherent instability of forward-looking forecasts and seeks to answer the question: "How can we ensure that the predictions we make today remain valid tomorrow, in the next quarter, or even in the coming year?"

Using machine learning is the instinctive first step. Why not let machines handle it and observe the results? If they can't solve it, one might simply conclude, "Oh well, that's that."

In the paper, I demonstrate that the predictive landscape can be much simpler and more straightforward when we consider the underlying mathematics. A key factor is the interaction among the variables. Are these variables independent? In most optimization scenarios, they aren't. For example, the required inputs might depend on the costs of raw materials. If the price of one material increases, there might be a cheaper alternative that can be just as effective. Essentially, the interdependencies of inputs and their characteristics, such as costs, offer a clear framework for relatively easy mathematical analysis, without the high computational costs and complexities associated with machine learning.

As a pioneer in High-Frequency Trading and Data Science in Finance, I am certainly not a Luddite. However, I firmly believe in using computing power wisely to produce meaningful results without falling into the pitfalls of overfitting or other limitations. Simply feeding data into a machine and observing the outcome is a possible approach, but it is neither the most elegant nor efficient. Basic Mathematics and Statistics are foundational to Optimization studies for good reason, so let's make use of them!