Can we create AI models to predict macroeconomic trends?
Not too long ago, weather predictions were hit or miss. The joke was that the TV weather forecaster was carrying an umbrella on what was to be supposedly a nice day. Today, we can accurately model weather patterns for several days into the future.
Similarly, given enough relevant data, including market indicators, human behavioral patterns, current events, and the recent "black swan" events, we should be able to model macroeconomics.
In the image above, I show a tiny fraction of the market indicator data I have used. It is free, up-to-date, and available for download in tabular format, mostly as Comma-separated Values (CSV or spreadsheets).
Markets fluctuate up and down. Being consistently better than the average, even by a few percent, in predicting the future would be a big benefit.
Markets fluctuate up and down. Being consistently better than the average, even by a few percent, in predicting the future would be a big benefit.
"Your goal is to be less wrong." - Elon Musk
For years, I have been discussing the concept of AI modeling with my friends based on thousands of macroeconomic data streams. Such modeling could give the architect of such a system enormous opportunities.
Now, the data, the software, and the hardware are within reach of that goal.
It is prevalent to hear business or political pundits speak of what amounts to a "mono-thematic bias" based on the human mind's minimal ability to grasp only a few dimensions of a given topic.
In his project, I use machine learning to combine thousands of complex data streams (think of each stream as a different spreadsheet) into a consistent and verifiable "oracle."
At the start, I employed easily accessible market indicators and data that influence the immediate macroeconomic changes such as manufacturing, unemployment, new housing permits, inventories, etc.
I used averages heavily as I realized that certain events build up to a change over periods, sometimes days, weeks, or months.
Of course, I realize that more than historical data is needed.
I am also interested in human behavior. Pundit and influencer opinions, even if wrong, definitely affect human behavior.
Markets, or rather large groups of people, behave illogically.
However, there are patterns in there in how an average Joe listens to the presidential speech, watches a few pundits argue, reads an article or two, talks to friends at work, and watches a few videos titled "Imminent crash in [insert any year]," hears that someone got laid off, and finally starts selling their stocks a few days later, making a colossal investment mistake.
Such modeling is difficult, but it is achievable.
It is prevalent to hear business or political pundits speak of what amounts to a "mono-thematic bias" based on the human mind's minimal ability to grasp only a few dimensions of a given topic.
In his project, I use machine learning to combine thousands of complex data streams (think of each stream as a different spreadsheet) into a consistent and verifiable "oracle."
At the start, I employed easily accessible market indicators and data that influence the immediate macroeconomic changes such as manufacturing, unemployment, new housing permits, inventories, etc.
I used averages heavily as I realized that certain events build up to a change over periods, sometimes days, weeks, or months.
Of course, I realize that more than historical data is needed.
I am also interested in human behavior. Pundit and influencer opinions, even if wrong, definitely affect human behavior.
Markets, or rather large groups of people, behave illogically.
However, there are patterns in there in how an average Joe listens to the presidential speech, watches a few pundits argue, reads an article or two, talks to friends at work, and watches a few videos titled "Imminent crash in [insert any year]," hears that someone got laid off, and finally starts selling their stocks a few days later, making a colossal investment mistake.
Such modeling is difficult, but it is achievable.
We regularly deduct human sentiment from Google searches, YouTube videos on the newest Apple or Tesla innovations, Twitter blasts, and blog posts. It is a known and proven technology.
This project is not meant to be a quick win.
This project is not meant to be a quick win.
I realize that every hedge fund is pouring millions into similar research.
Rather, I want it to be a gradual learning process leading to a recurrent retirement income. I still have a couple of decades to solve this complex problem.
I plan to add new data sources every few days and continue building the models until I get good at predicting something. I have used the word "something" on purpose. I am not sure if I will be able to predict the price of the S&P 500 company, but I am sure I will find patterns in some niche or index fund, and having a wizard-level insight might open doors, a path less taken.
This article and the accompanying open-source code are not meant to teach you about the market or how to code but to share my progress and receive positive feedback and cooperation.
Project Objectives
Market Indicators are collections of data points (think spreadsheets) reflecting the historical performance of a particular area of interest, for example:
There are hundreds, if not thousands, of such indicators.
I plan to add new data sources every few days and continue building the models until I get good at predicting something. I have used the word "something" on purpose. I am not sure if I will be able to predict the price of the S&P 500 company, but I am sure I will find patterns in some niche or index fund, and having a wizard-level insight might open doors, a path less taken.
This article and the accompanying open-source code are not meant to teach you about the market or how to code but to share my progress and receive positive feedback and cooperation.
Project Objectives
- to gather and analyze as many market indicators as possible
- to learn patterns and interactions in the data
- to predict market trends for a couple of days ahead
- to predict a particular index fund price
- to produce a reusable code and document the process
What are Market Indicators?
Market Indicators are collections of data points (think spreadsheets) reflecting the historical performance of a particular area of interest, for example:
- "S&P 500" index shows how major 500 stocks as a whole are performing
- "ISM Manufacturing" index shows how well the manufacturing industry is doing
- "GDP" (Gross Domestic Product) index shows how the country is doing
- etc.
There are hundreds, if not thousands, of such indicators.
Each country, state, county, and community has data sets that reflect some trend.
The spreadsheets and their graphing capabilities are the bread and butter of an individual's market analysis.
However, spreadsheets are good when comparing only a few indicators; the inputs become overwhelming when the data starts multiplying.
Why not use spreadsheets?
The spreadsheets and their graphing capabilities are the bread and butter of an individual's market analysis.
However, spreadsheets are good when comparing only a few indicators; the inputs become overwhelming when the data starts multiplying.
Please remember that an average person can remember about 7 numbers. Now, imagine daily data for the last few decades coming from a few hundred indications, multiplied by a couple of hundred regions, and adding sentiments from hundreds of sources. We can easily talk about millions, if not trillions, of inputs.
Today's computers can process many "tera" operations per second. Tera is 1,000,000,000,000.
Today's computers can process many "tera" operations per second. Tera is 1,000,000,000,000.
The point is clear.
The overwhelming advantage of machine learning, or Artificial Intelligence (AI), is apparent when we try to find subtle patterns in thousands of data flows.
The human brain (using spreadsheets) often fails to grasp the wealth of the information presented. Machine learning, on the other hand, can easily detect the patterns in massive datasets and derive a conclusion.
If I had chosen Python, I would have made my life easier.
However, considering a multi-year investment, I wanted to select the best language.
Here are a few of the characteristics of the Julia language:
Julia is modern, fast, elegant, multitasking, and does math extremely well. Of course, we are talking about the Julia programming language, not a person.
Unlike C/C++, it is a pleasure to read and write Julia.
The Julia language is designed at MIT specifically for scientific computing and machine learning.
Similar to C, it is fast.
Similar to Python, the Julia language is easy to learn.
The Julia language is designed for parallelism.
It is designed for distributed computing, as one day, I might decide to run hundreds of predictive models in parallel and not in sequence as I do now.
I am making changes to the project's code daily.
The logical approach and the code will surely diverge from any examples I could provide, hence the pun on Obi-Wan Kenobi's "Remember, Luke, use the source." In other words, study the actual code.
See the actual predictions.
I have provided a page on my GitHub where I posted the results of my predictions.
The rest of the project is documented at:
https://github.com/UkiDLucas/MarketIndicators.jl
Based on the feedback, I will be either motivated to write more or reconnect with my innate introvert.
Contact me via LinkedIn
https://www.linkedin.com/in/ukidlucas/
Please copy and paste below if you are referring to this article:
Uki D. Lucas. 2020. Market Indicators - ML project with Julia language.
https://uki.blogspot.com/p/market-indicators.html
Machine Learning Approach
The overwhelming advantage of machine learning, or Artificial Intelligence (AI), is apparent when we try to find subtle patterns in thousands of data flows.
The human brain (using spreadsheets) often fails to grasp the wealth of the information presented. Machine learning, on the other hand, can easily detect the patterns in massive datasets and derive a conclusion.
Why am I using the Julia language?
If I had chosen Python, I would have made my life easier.
However, considering a multi-year investment, I wanted to select the best language.
Here are a few of the characteristics of the Julia language:
Julia is modern, fast, elegant, multitasking, and does math extremely well. Of course, we are talking about the Julia programming language, not a person.
Unlike C/C++, it is a pleasure to read and write Julia.
The Julia language is designed at MIT specifically for scientific computing and machine learning.
Similar to C, it is fast.
Similar to Python, the Julia language is easy to learn.
The Julia language is designed for parallelism.
It is designed for distributed computing, as one day, I might decide to run hundreds of predictive models in parallel and not in sequence as I do now.
May the source be with you, Luke
I am making changes to the project's code daily.
The logical approach and the code will surely diverge from any examples I could provide, hence the pun on Obi-Wan Kenobi's "Remember, Luke, use the source." In other words, study the actual code.
See the actual predictions.
I have provided a page on my GitHub where I posted the results of my predictions.
The rest of the project is documented at:
https://github.com/UkiDLucas/MarketIndicators.jl
Based on the feedback, I will be either motivated to write more or reconnect with my innate introvert.
Contact me via LinkedIn
https://www.linkedin.com/in/ukidlucas/
Citation
Please copy and paste below if you are referring to this article:
Uki D. Lucas. 2020. Market Indicators - ML project with Julia language.
https://uki.blogspot.com/p/market-indicators.html
