As a frequent Kaggler (perhaps too frequent... it's a bit addicting, in a way I'm sure others on HN will understand), I was fairly intrigued to see this one pop up in the competition list a few days ago. Finance shops have tried their hand at Kaggle before, but I think they've normally been out of their domain. e.g. Two Sigma recently did a reinforcement learning game competition.
I'd caution the HN crowd not to expect production-level quant models out of this, like I'm seeing some doing in the comments already. Kagglers are excellent machine learning practitioners and the models that come out of many competitions are top-notch stuff, often making their way into research papers. But this is a short competition on limited data in a non-real-world scenario. The winning models will be very interesting educational exercises and probably wonderful recruiting material for Jane Street, but won't be the underpinnings of a new fund.
That said, I can't wait to see what comes out of this one. It ticks all of my competitive boxes :)
Mathematical analysis of financial markets is more celebrated when applied to relative valuation of different assets, rather than prediction of the market. Black-scholes, for example, applied calculus with an underlying no-arbitrage assumption to create a thriving market in option pricing, by giving traders a mechanism to reduce risk and thereby reduce bid offer spreads. Same in fixed income, mortgage, and credit market assets over the years.
The problem with predicting absolute levels, is that there is a game theoretic aspect which undermines any mathematical trading strategy as soon as it is public. optimal game theory trading strategies donât produce great results, and they are relatively trivial to identify. Instead strong profits in market long/short macro positions are mostly created by information advantages, which donât really make for interesting Kaggle competitions. For example, big profits in macro trading have historically been consistently achieved by front running customer orders, by building timing advantages on top of trading infrastructure, by funding research analysts that inspect operations on the ground, by lobbying for regulations that change market directions and so on.
Itâs very hard to tell if a best performing hedge funds that doesnât have an unfair advantage, that declares its only using quantitative strategies, is in fact just a statistical anomaly with a hollow narrative.
This!
>> The problem with predicting absolute levels, is that there is a game theoretic aspect which undermines any mathematical trading strategy as soon as it is public.
I took finance in Business school, coming from doing a lot of statistical analysis in a research lab. I hated my finance professors and there pseudo science. Pricing formulas work great until they don't. The problem is when they don't, they really don't, in a catastrophic way. Read "When Genius Failed." Real traders know this. But some economists and finance professors act like these mathematical models are describing a predictable physical phenomena.
To clarify, the hedge fund LTCM in "When Genius Failed", collapsed not because it relied on arbitrage 'pricing formulae', rather because it failed to properly execute arbitrage trades.
LTCM in being overly leveraged, relied on other market participants to maintain short term price alignment, which meant it was not arbitrage. Salomon's reduced its role as market-maker, maintaining short term price alignment, which increased short term price anomalies, and thus increased LTCM's vulnerability. The Asian financial crisis increased the frequency and extent of those pricing anomalies, and the subsequent Russian Default crisis did the same. Margin calls were made on LTCM that it couldn't cover, forcing them to close out of their positions at very unprofitable times of the trade strategy.
So I don't think "pseudo-science" is a great description for what those B-School profs are teaching. Rather the pricing formulae are just the beginnings of the financial theory you need to run arbitrage strategies, but they are not sufficient. You need to augment them with a broader picture of market dynamics and capital management, just like you'd need to learn about financial law, financial market technology, and a bunch of other stuff to run a successful market-making desk.
To add to this, unless your model is situated, and can purturb the market, it has no way of knowing what happens when you flex your muscle. I have a friend who did algorithmic trading professionally for a few years, and he said it was amazing to watch the data. Said he could see other bots come along and poke him, trying to look for weaknesses in his algorithm to exploit. I would expect a purely formulaic trader to underperform other traders who can take advantage of others. Itâs no different than how you have to win a rhoshambo turnament.
I've wanted to start learning about this for a while but I'm really not sure where to start. I have a degree in CS and Math so I'm not a total layman wrt the maths. Do you have any suggestions?
This might be a good starting point.
https://www.wiley.com/en-us/Paul+Wilmott+Introduces+Quantita...
What's your goal?
> relative valuation of different assets, rather than prediction of the market
But isn't prediction an inherent part of valuation?
Relative price prediction is an inherent part of valuation. You predict what the price of, say, a bond is given the price of the discount rate over the life of that bond. You are not predicting the absolute price of the bond, you are not able to predict if rates are going to go up or down. That's the appeal of arbitrage, you don't need to see the future, you make money no matter what if you see that a particular asset is 'out of whack', mis-priced, cheap, expensive, and you buy/sell it (and execute the appropriate arbitrage hedging strategy until maturity of the trade).
>But isn't prediction an inherent part of valuation?
Doesn't need to be. You have wealth you have to store somehwere because it is not being spent now. Can store it as equities, debt (including a bank account), Paper cash money under the mattress, gold bars, BTC, etc etc.
Let's say you have the list of potential places to store it and can get a price/value ratio but with a common term of "X" in the value
- A = 3.1/X
- B = 2.8/X
- C = 1.9/X
- D = 4.2/X
Yes, this is a massive simplification as it ignores diversification benefits and so on but there's the basics of the usefulness of valuation where you don't and can't know the value.
"prediction of the market" here means "Do you think the S&P500 is going up or down this year?" Warren Buffet completely ignores this, takes no view on it. Timing the market is very hard. Assuming the market will reflect the economy and the economy will grow because people work hard, are smart, we invent new things and population increases, then picking the best of what's available is the normal value investors route, a la Buffet as the most famous practioner.
Burton Malkiel's "A Random Walk Down Wall St." is still, IMHO the best summary of all the theory and practise of investing - excluding the quant funds like Renaissance. Maybe he has an update for them? Maybe it's also worth reading if it exists or maybe not. Don't know.
No idea if renaissance time the market or what.
edit: punctuation and the Malkiel recommendation.
This is highly dependent on the model in question. If you look at the parameters of the Black-Merton-Scholes model, there are assumptions embedded in the model that aren't necessarily predictions.
Also, prices aren't stationary. For an equity security, you are predicting the price for a company that is compounding capital over time. You can predict the price for the company at one point, the relative valuation for the company (for example, against peer group) may not change in one year but that company is investing their capital at X% so you get price growth.
The reason why relative valuation models are more effective is the same reason why most sports betting models use current odds as an input. Prices contain information but, in my experience, these methods aren't totally effective because they often miss important information about the company itself (big price moves happen because relative valuations are wrong). Value or quality appears to do fundamental work but is often woefully blind (for example, there are proven accounting issues with value strategies...does your average quant understand this? No. Have they ever read a set of accounts? No. They have no hope. None.)
Just imo, I think quant strategies are almost totally worthless beyond liquidity provision (even a strategy like front-running news in FX...humans do this better, and I know people who are still making tons of money doing this). I think there is massive value in that mode of analysis but the people who make the most are always going to be people who know the fundamentals better (I think firms like Marshall Wace that are doing this synthesis will move ahead) because that information is often not in the price at all.
> Also, prices aren't stationary
I'm not sure about the rest of your comment, but this is mathematically the correct reason why we don't predict absolute price levels.
And the reason why this is mathematically the correct reason is because for a non-stationary process, when you predict into the future, the variance tends to infinity which means taking expectation on any statistical model is useless since the variance is ridiculously wide
Yes, it's overwhelmingly unlikely that the winning model will actually be a competitive trading strategy.
Kaggle encourages a domain agnostic approach to modeling, in the sense that participants use sophisticated machine learning and statistical methods but typically have no domain expertise in the underlying data. This kind of approach to finance has historically performed poorly. [1]
Good quantitative trading is usually backed by a strong fundamental thesis and an interpretable model, which is obtained by cross-pollinating sophisticated math and statistics with domain expertise in some part of finance. That domain expertise might be in different kinds of assets, liquidity or market microstructure, but it's there.
$100k is cheap for Jane Street. If nothing else they have a new recruiting pipeline of people with demonstrable machine learning skills.
______________
1. I would also say this is a poor way to approach statistical analysis in most domains, and usually leads to spurious or overfit results. But the idea that you can just run a model and find patterns in pricing data is especially attractive and insidious.
> Kaggle encourages a domain agnostic approach to modeling, in the sense that participants use sophisticated machine learning and statistical methods but typically have no domain expertise in the underlying data.
Yes this is accurate and put very well. This is so much the case that if you have a strong background understanding of the field, the ML part can actually be picked up quite quickly or contributed by someone else. There are a few notable users who are both domain and ML experts and they tend to absolutely clean up in their field. I'm thinking of a couple of med students in particular who are formidable in every medical imaging competition.
"have no domain expertise in the underlying data. This kind of approach to finance has historically performed poorly"
I recently read 'The man who solved the market', about Jim Simons and Renaissance Capital. The way the book tells it, looking for patterns without seeking domain expertise (e.g. ignoring fundamental valuation of equities) is exactly what Renaissance did, and it worked out very well.
I can see why someone would characterize RenTech that way but it's not really fair to do so. There is a lot of mythos about how Simons hired computer scientists, mathematicians, signal processing and NLP experts, etc. When Mercer came over from IBM, he definitely contributed a significant amount of analytical expertise that was probably nonexistent in financial trading at the time (with the possible exception of the Ed Thorp diaspora). The astrophysicists RenTech hires every year bring new insights in ways to model and understand vast amounts of data with absurd dimensionality.
But all of this has to be utilized in the context of the data. The reality is that you're not going to develop a sophisticated options trading strategy without a strong understanding of what an option (and more generally, a derivative) is. You can't develop a viable statistical arbitrage strategy just by treating market microstructure as a blackbox signal to be solved with e.g. Fourier analysis. You can certainly find an edge in using fundamentally superior methods of analysis, but you still need to know what that data represents in the context of the market.
Don't be fooled: people working at firms like RenTech have a strong understanding of the underlying finance. It's just that they learned it on the job, because the ethos at these firms is that learning fundamental theory in math and statistics is harder than learning fundamental theory in finance. You don't have to take my word for it though. Read about one of the few strategies of RenTech's which has been publicized: https://www.bloomberg.com/opinion/articles/2014-07-22/senate.... Deutsche and RenTech didn't team up on this strategy (to fantastic success) by treating basket options as some kind of blackbox abstraction devoid of delta, gamma, theta and vega.
We should also remember that Jane Street is primarily an ETF market maker. Their main business isn't betting on prices of stocks or managing a portfolio.
I've only taken a quick look at the data, but the problem doesn't seem to be focused on their core competencies, but instead is much more general.
>I've only taken a quick look at the data, but the problem doesn't seem to be focused on their core competencies, but instead is much more general
How can you tell? All the features are completely anonymized.
They even say in the instructions:
Admittedly, this challenge far oversimplifies the depth of the quantitative problems Jane Streeters work on daily, and Jane Street is happy with the performance of its existing trading model for this particular question.
Any model superior to what Jane Street is running is worth vastly more than the prizes theyâre offering.
If you prove such a model out, get licensed (SEC, FINRA) and start soliciting to manage assets.
Disclaimer: Not investment advice. Not a lawyer, not your fiduciary.
>Jane Street has spent decades developing their own trading models and machine learning solutions to identify profitable opportunities and quickly decide whether to execute trades. These models help Jane Street trade thousands of financial products each day across 200 trading venues around the world.
>Admittedly, this challenge far oversimplifies the depth of the quantitative problems Jane Streeters work on daily, and Jane Street is happy with the performance of its existing trading model for this particular question. However, thereâs nothing like a good puzzle, and this challenge will hopefully serve as a fun introduction to a type of data science problem that a Jane Streeter might tackle on a daily basis.
Sounds like it's just for fun/recruiting rather than trying to crowd source new strategies -- I'm sure if they were looking to crowd source strats they'd pay a whole lot more than 40k for first place
This contest seems like the equivalent of the "inventor's hotline" infomercial. If it identifies one promising new approach that they can iterate on, it has probably paid for itself. It also serves as a good PR and recruiting tool. The prize is probably designed to bring in clever non-professionals. It's a win-win for Jane Street
The inventors hotline infomercial is a scam where they get you to pay for expensive patent filing, consulting, and marketing packages. They never intend to actually use any of the inventions.
If someone has a good idea, you don't want the idea, you want the person. If you take the idea, at best you'll split the market with the person who had the idea. At worst they'll iterate and you'll get nothing. Far better to find people who have the skills to develop an idea.
Having said that you also want to find the (vastly more in number) people who can take someone else's idea and actually implement it.
I sincerely doubt they think they'll get actionable ideas. It seems like a fun recruiting play from a company that takes pride in hiring non-traditional talent.
why would they need to pay more? the nature of smart people is to undervalue themselves and to not negotiate, so as long as smart people keep doing that other people can take advantage of that.
True, and I don't think they expect models that are superior to their own, I'd look at this as a hiring / marketing tool. Plus, even if you had a model that from a pure engineering standpoint was able to match Jane Street's approach, the model would not work without the wealth of proprietary (and expensive) data sources that Jane Street is sure to ingest, so you still couldn't just go out and do it yourself without some serious upfront investment first to get the same data. That is assuming all data they use is even available commercially, which I doubt as well. There are probably data sources that only become available to you through personal relationships with the right folks at the right places.
To me, this seems more like a funnel for recruiting
With an engineer phone screen and three on-site interviews, that's 4 hours of engineer time. $150/hr compensation per engineer, so cost is roughly 2x, $300/hr. So $1.2k to run a candidate through the pipeline post-initial-qualification.
To get one candidate and come out superior, acceptance rate should be 1%. (i.e. 99 failures). But if there are 50 leads from the program, and you convert a fifth, that's 10 candidates for a cost / successful recruit of $10k which means you have 10% acceptance rate to break even.
Hmm, back of the envelope seems to do all right as a strat. Relatively cheap. I recall the last time we were hiring, we projected cost per hire at $35k with the bulk of that actually being the recruiter referral fee.
I think you are significantly underestimating the cost per hour of jane street employees
Such competitions might have two goals in mind: recruiting and signal diversification. The recruiting angle is obvious.
Any alpha that is not fully correlated to existing alpha is worth its weight in gold for an organization with the size, sophistication and complexity of JS. That's part of the reason why efforts such as 2Sigma's Alpha Studio exist: https://alphastudio.com/
I'm sorry, if you could build a model to predict markets, why will you post in to Kaggle to get $40k in prize instead of applying this model to your own broker account?
It is much harder to turn a model into a profitable trading strategy than people realize. Apart from transaction costs, risk management and market impact there are also a lot of small operational details which can make or break your execution. One example I vaguely recall was that the details of how a specific foreign exchange conducted its closing auction could make a substantial difference to a strategy that involved executing there alongside other trading venues.
The payoff for getting these operational details right or wrong is massively asymmetrical. If you get everything right, you'll only do as well as your model lets you. But if you get anything wrong, you run a real chance of losing far more money than you could have hoped to make!
Even just validating your strategy on historical data (ie back-testing) is harder than it sounds. If you make a mistake that leaks information to the code you're testing, you can end up with a much rosier return and risk profile than you really have. Another way to lose money when you go put your model into action.
If you get over these challenges and run your strategy successfully for a while, other market participants are going to start adjusting against it and you have to adjust in turn. You can't just "set and forget".
I should note that I am far from an expert on any of this, though! I just know enough to not trade with serious moneyâmy real savings are all in index funds I don't touch, thank you very much :).
This is basically why Iâve never seriously considered doing it myself. I had a neat idea in about 2005 which I tested on historical data, and it beat buy-and-forget on every share I tried except for Google, and it only had one free parameter.
But, even if Iâd implemented it perfectly, and even if the algorithm has survived the financial crash, it wouldâve only worked if I could trade for free, and other people copying the algorithm would probably have made it stop working.
I believe what you are referring to is the fix. Foreign exchange markets, that I am aware of, do not have closing auctions.
I have heard of some quants trading foreign exchange markets, agreeing to trade at the fix with their counter-party, and not realising that traders often manipulate the fix resulting in the quant's strategy appearing not to work. It is almost comical (I worked in finance but not in FX, everyone knew this was going on for decades before the SEC starting fining people) that someone who managed money was making this error.
You are 100% correct about all the other stuff. Lots of issues with "production"...that is why financial firms employ traders/risk people/etc. Most people who trade themselves tend to go for lower-frequency strategies that they can implement personally. I actually don't think there are huge barriers, smaller investors have a huge advantage (when you trade at scale, the market moves against you) but you have to work with what you have and realise that you will get crushed if you try to replicate what someone with more money is doing.
Also, data. Data is expensive, and a huge fixed cost.
A âforeign exchangeâ not âforeign exchange marketâ
well because quant trading isn't about import xgboost, you need a sustainable infra to handle api failovers, bad data... not even going to mention risk management which is 50% of what quant trading is about. the data provided is anonymized but would probably be a mix of laggard measurements (moving averages, rsi...) and maybe some flow data... quant trading isn't really about finding "secret stuff" most profitable strats you can deploy can be based on stat-arb, basis trading or even just delta-neutral funding farming and such
It's a good question. The basic answer is not everyone has capital and risk, but they may have the time and intellect.
Mostly because itâs impossible to accurately predict the market - and this is just a competition to see who can build the best model.
HFT firms aren't trying to predict "the market" as a whole - just small eddies of it. A typical example of this is arbing names at the bottom of index fund rebalances. Speed is important mostly to make sure someone else doesn't hit the arb first.
By a similar argument to https://danluu.com/sounds-easy/ , no one will beat Jane Street in a weekend.
Jane Street's hiring standards exceeds FAANG's.
This is a hiring/branding strategy. Good luck to them.
Jane Street's interview was one of the funnest I've ever done. I failed but the questions were great!
> no one will beat Jane Street in a weekend
They have high hiring standard, but I suspect their 100s of smart engineers can't compete with the rest of the world.
I actually give it 48 hours before the top 3 equals what Jane Street can do in-house on this exact same dataset. A week before the reasonable plateau is reached, and a month or so before the absolute most information is squeezed out.
Yeah, I meant that Jane Street aren't doing this to find a solution that'll improve their own tech.
I expect in a constrained environment, someone on Kaggle will beat Jane Street's solution.
https://numer.ai is an entire hedge fund built around an anonymous ML prediction tournament. They solicit predictions, trade them, and reward the best performing ones. IIRC Theyâve paid millions in prizes over the last few years.
They also recently introduced Numerai Signals, where they pay for the performance of actual training data. So you can make money of providing datasets that perform well.
Isn't it pretty well known in the finance world that using stale public information to predict the market is a fool's errand?
Unless you have some kind of specialized non-public data (e.g satellite images of number of cars parked outside parking malls, number of cargo ships moving in and out), trying to predict the market with historical data does worse than "Just give me some monkeys, darts and a dart board".
Using purely historical price data it is harrowingly difficult. There are 130 anonymized features, so that's unlikely to be only price data. It could include information on the order book, correlated assets, fundamentals, vectorized/embedded text, etc.
Besides, I bet you can train monkeys to do (slightly) better than blindfolded random throwing. Even with public data (replace satellite images with Youtube mentions, or number of links moving into a company website) it is very possible to do better than average guessing on quite a lot of assets (especially smaller and newer markets).
Most hedge funds, even with specialized expensive non-public data, are not magical unicorns. Their quants really may just run a gradient boosting machine and leave it at that. Some hedge funds even prefer linear methods, because this lowers risk through lower variance. Such models can be beaten by experienced Kagglers for sure. For one, I did.
One thing we need to be clear about is that you're not aiming to be better than average. You're aiming to make a profit. There are probably hundreds of thousands of day traders, there are probably <100 market makers and tradingfirms (far less than that for a some specific products) and you'll probably find 99% of the day traders aren't making systematic profits. There are lots of strategies that are much better than average and still worse than putting your cash in a bank.
You can aim for both. If you just aim for profit, then you can get lucky with just average, or even random, betting. If you find a weighted coinflip (which is not impossible), provided by how many times you can flip that coin, you will see steady systematic profits. Of course, majority of day traders are getting owned by the big players, and they would do better doing more reasoned and long-term investments. Most day traders are not even using predictive models though.
Theres actually still money to be made in small scale strategies. Sophisticated funds are running billions. They cant focus on strategies that only work for 100-500k. This is where big returns can be made. Even warren buffet will say, if he was only managing 1 million, he would get 100% a year returns.
That doesn't make sense unless there are very few viable small scale strategies, at which point they'd probably be difficult to identify. Your assertion might have been true before computers were able to help someone manage many strategies simultaneously.
Granted, a typical Kaggle metagame-that-is-technically-against-the-rules is to use data from outside the dataset, which is one of the reasons winners have to be validated.
From: kaggle.com/c/jane-street-market-prediction/overview/code-requirements
"Freely & publicly available external data is allowed, including pre-trained models"
Generally this is allowed if you publish the data you're bringing in. They even create a sponsored thread for it in most competitions.
>Unless you have some kind of specialized non-public data (e.g satellite images of number of cars parked outside parking malls, number of cargo ships moving in and out)
Planet labs will sell you all of that data, in case people reading along here are curious.
If I build a model that actually works well, I'm using it to get rich, patent it, and sell it to the company for a lot more than $100k.
Maybe or maybe not. You may need to build up your trading infrastructure first, which entails among other things low-latency connectivity to different venues, negotiate good deals with brokers to get low trading fees etc. If it were that simple, all the quants would be working for themselves. Trading is not just about having good prediction. Also if you publish/share your algorithm, people will copy it and it will lose its edge.
If you are really good so that whereas others can only predict the future 0.1s but you can predict the future 5s with the same accuracy, then sure, you could trade from home over the Internet, and if you are much more accurate than others, especially when the market moves a lot, you don't need low fees to be competitive.
"Also if you publish/share your algorithm, people will copy it and it will lose its edge."
That's why I list patenting it after getting rich.
The quants don't work for themselves because they're number crunchers and need the financial knowledge that the trading/portfolio managers have. Either way, the main reasons they don't work for themselves is risk and access to capital.
IDK who will still buy a patent for the trading algorithm knowing that it's publicly available and probably not so competitive anymore.
This would be much more interesting if the features weren't anonymized.
At this point this is just a widest/deepest neural net competition on some unknown bunch of features.
100% this. Anonymised features remove any hint of lateral thinking that is the essence of science, and just focuses on the pure numbers. This is dangerous. One of my data scientists once ran up a $2000 bill on training a CNN on what was essentially a speed = distance/time calculation, and was really proud when he got 92% accuracy. A real facepalm moment.
I was just about to post the same thing. I'm a statistician in my day job and the raw math is only one part of building a model. Human judgement (while often flawed) can be key to improving actual model performance.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.