Knowledge Base
A collection of technical articles covering trading protocols, networking, latency, and machine learning.
BATS Binary Order Entry
BATS Binary Order Entry is a high-performance trading protocol modeled on FIX but adapted for simplicity and ultra-low latency.
Read articleCan two applications listen to the same port?
The short answer is "no, not on the same host." The longer answer is that this is by design, the basic rationale being consistency.
Read articleDifference between classification and clustering in data mining?
In data mining, classification is a task where statistical models are trained to assign new observations to a "class" or "category" out of a pool of candidate classes; the models are able to different…
Read articleDifference between Linear Regression and Logistic Regression
How Linear Regression and Logistic Regression differ in the functions they use, the outputs they produce, and the prediction problems each one is built to solve.
Read articleDifferences between Training, Validation, and Test Set in Machine Learning
When tackling a supervised machine learning task, the developers of the machine learning solution often divide the labelled examples available to them into three partitions: a training set, a validati…
Read articleFIX Protocol
The FIX protocol is widely used across all asset classes, and supported by most markets. It is the workhorse of communications between modern trading systems.
Read articleHow accurately can latency be measured?
The accuracy of a latency measurement derives directly from that of the timestamps used to construct it, which in turn is determined principally by two factors: a) the fidelity with which the timestam…
Read articleHow can I check the status of a given port on a remote host?
First, it's worth clarifying that by "status" we understand the question to be asking whether the port is open or not; that is, whether there is an application listening to that port.
Read articleHow can I create a self-signed certificate with openssl?
The simplest way to create a self-signed certificate is to use OpenSSL with the following one-liner: It is often useful to create a single .pem file containing both the key and the cert: These steps a…
Read articleHow can I simulate delayed and dropped packets in Linux?
The standard way to delay and drop packets in Linux is with the netem scheduling policy; this can be applied to any network device, whether an interface or a bridge, using the tc command from the ipro…
Read articleHow can TCP ACKs be used to measure latency to a server?
TCP ACKs can be used to measure the round-trip time to a TCP receiver, and they can do so very accurately: since ACKs are generated in the ISR (interrupt service routine), they involve only the lowest…
Read articleHow do I free up a TCP/IP port?
To free up a TCP/IP port that your application application has previously bound to, either call close() on the listening socket, or exit the application.
Read articleHow do Multiple TCP Clients Connect Simultaneously to a Single Port on a Server?
TCP connections are identified by a server TCP address and port and a client TCP address and port.
Read articleHow do you get Chrome to accept a self-signed certificate?
The following procedure, based on an answer provided by user: kgrote, works for Chrome 68 on Windows 10: Navigate to the site with the cert you want to trust, and click through the usual warnings for…
Read articleHow does a Genetic Algorithm work?
A step-by-step walkthrough of a genetic algorithm's generational cycle, from a random starting population through selection, crossover, and mutation to the single fittest surviving solution.
Read articleHow does one filter MAC addresses using tcpdump?
tcpdump supports the "ether" qualifier to specify ethernet addresses in the standard colon-separated format.
Read articleHow does one use tcpdump to capture incoming traffic?
The most reliable option is to use the -Q option as follows: The -Q option may not be supported on all platforms, and an alternative is to use equivalent logic in BPF (Berkeley Packet Filter) syntax,…
Read articleHow does wireshark annotate some packets with "tcp segment of a reassembled pdu"?
Briefly, Wireshark marks TCP packets with "TCP segment of a reassembled PDU" when they contain payload that is part of a longer application message or document that is completed in a later packet.
Read articleHow is latency analyzed and eliminated in high-frequency trading?
Latency is eliminated by making changes to the trading system software or infrastructure, and there is a wide variety of such changes that can be implemented.
Read articleHow is latency defined in high-frequency trading?
Latency is fundamentally the duration of a process, and can be quantified as the difference between the time at which the process starts and the time at which it ends.
Read articleHow is latency measured in high-frequency trading?
The latency of a process is measured by gathering timestamps reflecting the times at which the cause (e.g. price update) and effect (e.g. order placement) occur, and subtracting them.
Read articleHow to convert between different security certificate formats
There are many variants of these questions, asking how to convert TLS (technically X.509) certificates between various formats. To be stored in a file, a certificate must be encoded.
Read articleITCH Protocol
ITCH is an ultra-low latency protocol for accessing Market Data. ITCH has been developed to maximize performance and to meet the requirements of latency sensitive trading strategies.
Read articleNASDAQ TotalView ITCH
NASDAQ's full-depth market-data feed provides ultra-low latency and high-performance visibility of NASDAQ's equities markets.
Read articleNYSE Pillar
ICE NYSE's common technology platform modeled on FIX but with a focus on performance and ultra-low latency.
Read articleOUCH Protocol
OUCH is a high-performance trading protocol developed by NASDAQ with a focus on simplicity and ultra-low latency.
Read articleOverfitting, Variance, Bias and Model Complexity in Machine Learning
Comparing model error on training and validation data reveals whether a model is too simple or too complex, and where the sweet spot between bias and variance lies.
Read articleThe role of bias in Neural Networks
What the bias term does in a neural network, why it shifts the activation function, and why models need it, shown with and without bias.
Read articleWhat algorithm for a tic-tac-toe game can I use to determine the "best move" for the AI?
How the alpha-beta (minimax) search algorithm lets a tic-tac-toe AI weigh every possible move and pick one that guarantees at least a draw.
Read articleWhat are Naïve Bayes classifiers?
A step-by-step guide to how Naïve Bayes classifiers turn Bayes' theorem into predictions, worked through with a real network-failure example.
Read articleWhat are OLTP and OLAP and the difference between them?
These are terms relating to different kinds of business information processing systems.
Read articleWhat are the relative merits of TCP and UDP in high-frequency trading?
In general, the relative merits of TCP and UDP in high-frequency trading are the same as they are for most low-latency applications: on one hand, UDP has lower raw latency, by virtue of how much simpl…
Read articleWhat causes an IOC order to be sent at an unmarketable price?
There are many possible reasons that an order might be sent at a price that is not available in the market, some arising from the infrastructure and some more business-related.
Read articleWhat causes IOC orders to be canceled?
An immediate-or-cancel (IOC) or a fill-or-kill (FOK) order will be canceled if it arrives at the matching engine when there is no liquidity on the opposite side of the order book at, or better than, t…
Read articleWhat is a False Positive Rate?
How the false positive rate, the true positive rate and the confusion matrix work together to measure how well a classifier separates normal data from anomalies.
Read articleWhat is a Genetic Algorithm?
Genetic Algorithms are evolutionary search and optimisation methods that evolve populations of candidate solutions toward the fittest answer. Here is how the evolutionary cycle works.
Read articleWhat is a TCP Connection Refused?
In general, connection refused errors are generated during a connect system call when an application attempts to connect using TCP to a server port which is not open.
Read articleWhat is a TCP Connection Reset by Peer?
An application gets a connection reset by peer error when it has an established TCP connection with a peer across the network, and that peer unexpectedly closes the connection on the far end.
Read articleWhat is a TCP Reset (RST)?
When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset packet back on the same connection.
Read articleWhat is Anomaly Detection?
In many cases, malicious attacks initially manifest as unusual behaviors of user accounts, system activities on the network, or network traffic patterns.
Read articleWhat is Listening on a TCP Port?
To find what is listening on a TCP/IP port, open a terminal and go to a shell command-line, and use lsof, a command that LiSts Open Files (TCP ports are considered a type of file in UNIX).
Read articleWhat is the best way to describe latency?
The latency of an individual transaction or process is best described by a single number, namely the duration of the process in an appropriate unit.
Read articleWhat is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow?
How TensorFlow's two max-pooling padding modes differ, from their output-size formulas to a worked 4x3 example run with each option.
Read articleWhat is the Highest TCP Port Number Allowed?
The highest TCP port number is 65,535. The TCP protocol provides 16 bits for the port number, and this is interpreted as an unsigned integer; all values are valid, apart from 0, and so the largest por…
Read articleWhat is the largest safe UDP Packet Size on the Internet?
This question, in particular the word "safe" is somewhat ambiguous.
Read articleWhat is the lowest tick-to-order (or tick-to-trade) latency achievable without the use of FPGAs?
Without FPGAs, the lowest tick-to-order latency can be achieved by a server directly consuming multicast market-data, handling the feed internally, making a trade decision, and emitting an order using…
Read articleWhat is the maximum packet size for a TCP connection?
This question is also somewhat ambiguous, but for a different reason: the size of IP packets that a TCP stack uses to send data is chosen by the stack itself, and the user or application writing to th…
Read articleWhy does tcpdump not render ASCII packet data in a human readable format when tcpflow does?
tcpdump and tcpflow are two very different tools with very different purposes. It is true that both capture TCP/IP packets, or read them from pcaps, analyze them, and produce text output.
Read article