How to reject connections before TSL handshake?

[carenot]

My load test shows that when max_sessions is reached, new clients are rejected, but only after they've connected and completed TLS handshake.

This is not ideal as it still puts load on the LS server and handshake pool.

Is there a way to reject the connection straight away? Like sending a connection reset?

Please note that I just want to reject connection (i.e., create session requests). My LS deployment listens on two ports, one used for connection and the other for control requests. So I want to reject requests from one port (the connection port) ONLY.

A related question is what happens when handshake pool queue is full? Will this
a) block the server pool threads (because they hand over handshake task to handshake pool?), eventually fulling both queues, or
b) task is dropped and request eventually timeout on client side?

[DarioCrivelli]

About the handshake pool queue:
When you say that it is full, I suppose you refer to the <handshake_pool_max_queue> setting.
In this case, backpressure is put not on the "server" thread pool but directly on the listening socket, which stops listening until the size falls below the limit.
However, no tasks are dropped and no connection requests are actively refused. Only the connection requests that can't be enqueued in the configured backlog will be discarded; but this is left to the system, so I'm not sure of the effect on the clients.

In your case, where the create_session requests have a dedicated port, you can leverage a non-documented flag, <CHECK_SERVER_QUEUE>, and set it to N in the <http_server> or <https_server> blocks related to the ports not used for create_session.
Obviously, the preconditions for the leverage of the <CHECK_SERVER_QUEUE> setting can only be ensured by the application.

Indeed, something similar could be devised for the <max_sessions> limit; your request makes sense.
The hit of the limit might cause either immediate refusal of the requests (as you suggest) or block of the accept loop (as done for the max_queue case).
Unfortunately, this is not currently available but we will evaluate whether to introduce it in a future release.