When developing trading strategies, backtesting has become a routine step and is critical for validating future profitability. Historical data sets provide the input for backtesting, allowing traders and quantitative researchers to understand how their strategies would perform in real-world scenarios. There are many sources of historical market data in the industry, often intertwined with vendor-specific database technologies. As technology in the real-time market data space has evolved, the need for historical data has grown, and the technical details have become more sophisticated. An increasing demand for “raw” data capture has emerged, offering files of data stored in PCAP format, recorded from a network exactly as the exchange system sent it.
For nearly a decade, Pico has built a back catalogue of historical market data, storing these PCAP files split by exchange multicast groups exactly as they would have been distributed by the exchange. Pico clients can test their electronic trading strategies and new algorithms using true real-world scenarios, employing the same feed handlers and technology used in production. There is less need to work with alternative database technology.
Corvil Analytics - Market-Leading Capture Technology
At Pico, we capture exchange market data PCAPs using our market-leading Corvil network analytics solution, combined with Pico’s purpose-built ultra-low latency network and colocation community. Corvil Analytics is widely trusted by the world’s leading exchanges and the front office community as the market leader in network capture and analytics.
Co-located Timestamps
Corvil Analytics is installed in many of the exchange colocation facilities that Pico utilizes. This is a critical component of the design for Pico’s PCAP captures. Because Pico operates in the colo data centers, it enables us to capture data directly from the exchange’s handoffs and timestamp the data in nanoseconds, indicating the time the message was received within the colo. As a result, the timestamps in the files directly reflect real-life colo deployments, providing Pico customers with reassurance that they are replicating real-world scenarios for their algo testing.
Packet Loss
Pico captures four copies of multicast data across two resilient Corvil devices – the A and B feeds, twice! When replaying these files, it enables customers to perform arbitration on the data themselves as part of their backtesting, allowing for the replication of another real-world scenario. In a typical production trading environment, customers would need to process both the A and B feeds as part of their trading feed handler to simulate what happens if and/or when packets go missing. A significant benefit of Corvil Analytics is the reduction of packet loss during the capture process, something particularly challenging and important in today’s bursty and high-volume markets. Despite this technology advantage, packets can go missing in networks due to the nature of multicast, and our capture solution does not mask that; instead, it provides extra copies of the data to allow for arbitration.
Storing Data As Provided
In addition to capturing the A and B feeds, we also store data according to the exchange’s multicast channel definitions. This ensures that customers' backtesting can be easily transferred from concept to production without the need to re-engineer any code, as the data provided by Pico will further align with the real-life exchange format. Again, the production version of the feed handler does not need to differ from the backtesting version, simplifying the process.
In summary, Pico’s market-leading capture solution and purpose-built ultra low-latency network ensure we capture data as it would have been seen in real time during trading events. If you are looking for high-quality market data history or recordings provided on an ongoing basis, contact us to learn more: https://www.pico.net/contact-sales/.