What are the relative merits of TCP and UDP in high-frequency trading?

In general, the relative merits of TCP and UDP in high-frequency trading are the same as they are for most low-latency applications: on one hand, UDP has lower raw latency, by virtue of how much simpler its stack is compared to that of TCP. On the other, UDP provides none of the acknowledgement and retransmission that TCP does, nor any of its mechanisms to coordinate speeds between senders and receivers, both of which are immensely valuable for reliable communication. The lower latency afforded by UDP is very attractive in the context of HFT, but it comes at a cost.

In practice, the options for choosing between TCP and UDP in HFT are very limited: the market-data feeds that one would consume to engage in HFT are delivered over multicast UDP, while all exchange or broker interfaces for sending orders are TCP based. In a distributed trading system, or one using an external feed-handler, there is a choice between TCP and UDP for the middleware to connect the components, and the above trade-off between latency and reliabilty is relevant. However the very lowest-latency trading systems will be collapsed onto a single server, with any communications between internal software components best mediated via shared memory.

It is worth noting that FPGAs can reduce latency to the bare physical minimum even when using TCP. For example, a TCP packet can be partially assembled, with most of a full protocol message embedded in it, in the NIC (network interface card). Once a trade decision is made in software, only the final details of the order need to be transferred to the NIC where the FPGA completes the packet and any checksums needed, and transmits it. In this way, a fully standards-compliant TCP packet can be sent to an exchange with the lowest possible latency.