Alright – I think I finally figured out how to get market depth data by collecting it, but I have to wait until Sunday night to check that it is streaming the data to disk. This data appears far too expensive for an individual to buy, so your best option is to find a way to collect it. This approach _should_ work for cTrader if your broker supports it, and will write data real-time to disk, or can be modified to push updates to a stream for use as real-time observation. Order-book data is an excellent target for HFT strategies.

Published Oct 19 – This is untested currently as markets are closed – I’ll give it a run and update this if it’s broken.

Market depth data provides a valuable window into the supply and demand dynamics of financial markets. By collecting and analyzing this data, traders and researchers can gain insights that go beyond what’s possible with just price and volume information. In this article, we’ll explore how to use market depth data for automated trading strategies and machine learning applications.

Collecting Market Depth Data

To begin working with market depth data, we first need to collect it. We’ve developed a cBot (a trading robot for the cTrader platform) that collects market depth data at regular intervals, saving it to a CSV file. This data includes:

• Timestamp
• Bid prices and volumes for multiple levels
• Ask prices and volumes for multiple levels

The Market Depth Data Collection cBot

Here’s the cBot script we’ll be using to collect market depth data:

using System;
using System.Linq;
using System.Text;
using System.IO;
using cAlgo.API;
using cAlgo.API.Internals;

namespace cAlgo.Robots
{
    [Robot(TimeZone = TimeZones.UTC, AccessRights = AccessRights.FullAccess)]
    public class MarketDepthDataCollector : Robot
    {
        [Parameter("Symbol Name", DefaultValue = "EURUSD")]
        public string TargetSymbolName { get; set; }

        [Parameter("Depth Levels", DefaultValue = 4)]
        public int DepthLevels { get; set; }

        private MarketDepth marketDepth;
        private string outputPath = "/tmp/market_depth_data.csv";

        protected override void OnStart()
        {
            marketDepth = MarketData.GetMarketDepth(TargetSymbolName);
            marketDepth.Updated += OnMarketDepthUpdated;

            // Initialize CSV file with headers
            StringBuilder header = new StringBuilder("Timestamp");
            for (int i = 1; i <= DepthLevels; i++)
            {
                header.Append($",Bid{i},BidVolume{i},Ask{i},AskVolume{i}");
            }
            File.WriteAllText(outputPath, header.ToString() + "\n");
        }

        private void OnMarketDepthUpdated()
        {
            StringBuilder line = new StringBuilder(Server.Time.ToString("yyyy-MM-dd HH:mm:ss.fff"));

            var bids = marketDepth.BidEntries.Take(DepthLevels).ToList();
            var asks = marketDepth.AskEntries.Take(DepthLevels).ToList();

            for (int i = 0; i < DepthLevels; i++)
            {
                if (i < bids.Count)
                {
                    line.Append($",{bids[i].Price},{bids[i].VolumeInUnits}");
                }
                else
                {
                    line.Append(",,");
                }

                if (i < asks.Count)
                {
                    line.Append($",{asks[i].Price},{asks[i].VolumeInUnits}");
                }
                else
                {
                    line.Append(",,");
                }
            }

            File.AppendAllText(outputPath, line.ToString() + "\n");
        }

        protected override void OnStop()
        {
            if (marketDepth != null)
            {
                marketDepth.Updated -= OnMarketDepthUpdated;
            }
        }
    }
}

This cBot does the following:

1. Connects to the market depth data stream for a specified symbol.
2. Listens for updates to the market depth.
3. When an update occurs, it records the timestamp, bid prices, bid volumes, ask prices, and ask volumes for a specified number of levels.
4. Writes this data to a CSV file for later analysis.

You can customize the symbol, number of depth levels, and output file path according to your needs.

Running the cBot and Accumulating Data

To accumulate enough data for meaningful analysis and model training, you should run this cBot for an extended period. Here are some guidelines:

1. Short-term strategies: For intraday or high-frequency trading strategies, collect data for at least 1-3 months. This should provide enough samples to capture various market conditions.
2. Medium-term strategies: For swing trading or daily strategies, aim for 6-12 months of data. This will help capture seasonal patterns and longer-term market cycles.
3. Long-term strategies: For strategies operating on weekly or monthly timeframes, collect 2-5 years of data to ensure you’ve captured a wide range of market conditions.

Remember, more data is generally better, but you’ll need to balance this with storage constraints and the computational resources required for processing larger datasets.

Preprocessing and Feature Engineering

Once you’ve collected a substantial amount of data, the next step is to preprocess it and engineer features that can be used in your trading strategies or machine learning models. Here are some ideas:

1. Order book imbalance: Calculate the ratio of bid volume to ask volume at each level. This can indicate potential price direction.
2. Price pressure: Compute the volume-weighted average price (VWAP) of the order book and compare it to the current market price.
3. Spread analysis: Track the bid-ask spread over time and at different levels of the order book.
4. Liquidity measures: Calculate metrics like market depth (total volume available within a certain price range) or market width (price range containing a certain percentage of volume).
5. Order flow imbalance: Track the rate of change in bid and ask volumes to identify potential buying or selling pressure.
6. Volatility indicators: Use the variability in order book depth or spread to create custom volatility measures.

Applications in Automated Trading

Market depth data can enhance various types of trading strategies:

1. Liquidity-seeking algorithms: Use order book data to find the best execution prices and minimize market impact.
2. Mean reversion strategies: Identify temporary imbalances in the order book that might lead to short-term price reversals.
3. Momentum strategies: Detect strong buying or selling pressure that could lead to sustained price moves.
4. Market making: Optimize bid-ask quotes based on the current state of the order book.
5. Risk management: Adjust position sizes or stop-loss levels based on current market liquidity.

Machine Learning Applications

Market depth data provides rich input for machine learning models. Here are some potential applications:

1. Price movement prediction: Train models to predict short-term price movements based on current order book state.
2. Liquidity prediction: Develop models to forecast future market liquidity, which can be useful for optimizing trade execution.
3. Anomaly detection: Use unsupervised learning techniques to identify unusual patterns in the order book that might indicate large incoming orders or potential market manipulation.
4. Reinforcement learning: Create agents that learn optimal order placement strategies by interacting with a simulated order book environment.
5. Clustering: Group similar order book states to identify recurring market regimes or conditions.

Choosing the Right ML Approach

The choice of machine learning approach depends on your specific goals:

1. Supervised learning (e.g., random forests, neural networks) is useful for predicting specific outcomes like price movements or liquidity levels.
2. Unsupervised learning (e.g., k-means clustering, autoencoders) can help discover patterns or structure in order book data.
3. Reinforcement learning is suitable for developing adaptive trading strategies that can optimize execution in changing market conditions.
4. Deep learning approaches like LSTMs or transformer models can be effective for capturing complex temporal dependencies in order book dynamics.

Challenges and Considerations

Working with market depth data presents several challenges:

1. High dimensionality: Order book data can be high-dimensional, especially if you’re collecting many price levels. Dimensionality reduction techniques may be necessary.
2. Noisiness: Market depth data can be noisy, with many updates that don’t lead to significant price movements. Effective filtering and feature engineering are crucial.
3. Non-stationarity: Financial markets are non-stationary, meaning statistical properties change over time. Models need to be regularly retrained or designed to adapt to changing conditions.
4. Computational requirements: Processing and modeling large amounts of order book data can be computationally intensive. Efficient data structures and algorithms are important.
5. Look-ahead bias: Be careful not to use future information when preprocessing data or training models for historical backtesting.

Conclusion

Market depth data offers a wealth of information for developing sophisticated trading strategies and machine learning models. By collecting this data with our cBot and applying the techniques discussed in this article, you can gain valuable insights into market microstructure and potentially develop more effective trading algorithms.

Remember that working with market depth data requires careful consideration of data quality, feature engineering, and model selection. Start with simpler approaches and gradually increase complexity as you gain more experience with the data. Always thoroughly backtest and paper trade any strategies before considering real money deployment.