What is NTLM Authentication?

NTLM Authentication is a challenge-response protocol developed by Microsoft. It operates at the HTTP connection layer rather than the application layer. When a client requests a protected resource, the server responds with a 401 and a WWW-Authenticate: NTLM header. The client then initiates a three-message handshake: Negotiate (Type 1), Challenge (Type 2), and Authenticate (Type 3).

Because the server associates the authentication state with the underlying TCP connection, the client must use HTTP Keep-Alive. If the connection drops, the authentication state is lost.

Most scraping scripts are built around stateless requests. You fire a GET, receive the HTML, and close the socket. NTLM violently rejects this pattern. If you send the Type 1 message on Socket A, and the Type 3 message on Socket B, the server will reject the authentication because Socket B has no associated challenge nonce.

This means your HTTP client must be explicitly configured to reuse connections, and any intermediate proxies must not sever or multiplex the TCP stream during the handshake.

Out of the box, standard libraries like Python's requests, Node's axios, or Go's net/http do not handle NTLM. You must use wrapper libraries. In Python, requests-ntlm hooks into the authentication dispatch process to automatically perform the handshake when a 401 is encountered. In Node, libraries like httpntlm are required. Headless browsers handle it natively via their network interception APIs.

We maintain dedicated worker pools for legacy B2B targets. When a pipeline is flagged as requiring NTLM, we bypass our standard stateless proxy rotation. Instead, we allocate a pinned residential or datacenter IP, establish a persistent TCP connection, complete the NTLM handshake, and then stream the extraction jobs through that authenticated socket until the server forces a teardown. This guarantees zero dropped nonces and maximizes throughput.

NTLM is technically deprecated by Microsoft in favor of Kerberos. In modern IIS setups, you will often see WWW-Authenticate: Negotiate instead of just NTLM. This uses SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) to try Kerberos first, and fall back to NTLM if the client isn't part of the Active Directory domain. For external scrapers, Kerberos is usually impossible to negotiate, so the fallback to NTLM is what actually executes.