Here are some further points that explain when data is segmented and the benefits of segmented messages:
- Segmenting is predominantly for initial image – we would not expect it to kick in for a consumer that is processing updates
- Segmentation is currently size based so you’d only receive segmented message for ~1000 markets or more.
- We write segments until the socket is blocked – i.e. if a client is consuming quick enough then there is no additional latency.
- With segmentation it is much easier for a client to read data faster – and this is really the limiting factor (and segments are there to help as the server can be sending more data while the client is processing the block of data that it last received and decoding it).