问题描述

线上部署的代码，部署在测试环境突然抛出接口请求Read time out的异常。查看线上日志，接口请求正常。重新启动，部署测试环境代码都没有效果，接口还是必现Read time out。

原因分析：

1. 排查网络原因

直接在测试环境服务器上执行使用curl命令执行http请求，接口正常返回，所以排查网络，第三方服务的原因。

2. 网络抓包

解决这类问题，最简单的方法就是抓一个正常的http请求包，抓一个Read time out异常的http请求包，对比两个的报文。但是事与愿违，容器环境禁用tcpdump命令，也就抓不了包。

3. 本地代码debug

启动本地代码连接测试环境配置，再出发相同的请求，接口返回正常。

4. 调整日志等级

项目统一使用spring cloud体系微服务架构，将日志等级该为debug，日志会打印一些http请求信息。观查到报错前的http请求日志如下：

2022-12-05 09:47:21.209 DEBUG 1 --- [http-nio-8080-exec-44] s.n.www.protocol.http.HttpURLConnection  : [TxId : xxx-1^1670143309480^7363 , SpanId : 1300305726613613085] sun.net.www.MessageHeader@11ef701319 pairs: 
{GET /xxxx/audiorid?recordId=9776d4f1-8b89-488a-b00c-cd430f39c003&recordDate=1670146625251 HTTP/1.1: null}
{Accept: application/octet-stream, application/json, application/*+json, */*}
{System-Id: xxxapi}
{sign: xxxxxxxxxxxx}
{Sign-Type: RSA}
{Sign-Timestamp: 1670204841194}
{Content-Type: application/x-www-form-urlencoded}
{Content-Length: 135}
{request-source-app: xxxapi}
{Pinpoint-TraceID: xxx-1^1670143309480^7363}
{Pinpoint-SpanID: 3740963636577534541}
{Pinpoint-pSpanID: 1300305726613613085}
{Pinpoint-Flags: 0}{Pinpoint-pAppName: xxxapi}
{Pinpoint-pAppType: 1210}
{Pinpoint-Host: xxx.com}
{User-Agent: Java/1.8.0_231}
{Host: xxxxx.com}
{Connection: keep-alive}

因为本接口Method是GET，Content-Length应该为0，但是日志缺显示为135

5. 排查代码

http代码是基于restTemplate封装抽象统一实现的，以下是执行get请求的抽象方法：

public String executeGet(String systemName, String requestUrl, Map params, Map headerParams){HttpEntity> entity = RestHttpUtils.getEntity(params, headerParams);return execute(systemName, requestUrl, HttpMethod.GET, entity, params);
}

其中装换entity有点问题

public static HttpEntity> getEntity(Map params, Map headerParams){MultiValueMap mvm = getMultiValueMap(params);HttpHeaders headers = getHeaders(headersParams);return new HttpEntity>(mvm, headers);
}

在HttpEntity中，param被赋值给body，get请求中的param应该是queryParam，不应该在body中。现在赋值到Body，那么Content-Length就会有值。

public HttpEntity(@Nullable T body, @Nullable MultiValueMap headers) {this.body = body;this.headers = HttpHeaders.readOnlyHttpHeaders(headers != null ? headers : new HttpHeaders());}

Content-Length在restTemplate#doExecute中进行赋值

protected  T doExecute(URI url, @Nullable HttpMethod method, @Nullable RequestCallback requestCallback,@Nullable ResponseExtractor responseExtractor) throws RestClientException {Assert.notNull(url, "URI is required");Assert.notNull(method, "HttpMethod is required");ClientHttpResponse response = null;try {ClientHttpRequest request = createRequest(url, method);if (requestCallback != null) {requestCallback.doWithRequest(request);}response = request.execute();handleResponse(url, method, response);return (responseExtractor != null ? responseExtractor.extractData(response) : null);}catch (IOException ex) {String resource = url.toString();String query = url.getRawQuery();resource = (query != null ? resource.substring(0, resource.indexOf('?')) : resource);throw new ResourceAccessException("I/O error on " + method.name() +" request for \"" + resource + "\": " + ex.getMessage(), ex);}finally {if (response != null) {response.close();}}
}

具体见requestCallback.doWithRequest(request)，有body有值，所以Content-Length 没有赋值为0，而是在converter中序列化body并赋值。

public void doWithRequest(ClientHttpRequest httpRequest) throws IOException {super.doWithRequest(httpRequest);Object requestBody = this.requestEntity.getBody();if (requestBody == null) {HttpHeaders httpHeaders = httpRequest.getHeaders();HttpHeaders requestHeaders = this.requestEntity.getHeaders();if (!requestHeaders.isEmpty()) {requestHeaders.forEach((key, values) -> httpHeaders.put(key, new ArrayList<>(values)));}if (httpHeaders.getContentLength() < 0) {httpHeaders.setContentLength(0L);}}else {Class requestBodyClass = requestBody.getClass();Type requestBodyType = (this.requestEntity instanceof RequestEntity ?((RequestEntity)this.requestEntity).getType() : requestBodyClass);HttpHeaders httpHeaders = httpRequest.getHeaders();HttpHeaders requestHeaders = this.requestEntity.getHeaders();MediaType requestContentType = requestHeaders.getContentType();for (HttpMessageConverter messageConverter : getMessageConverters()) {if (messageConverter instanceof GenericHttpMessageConverter) {GenericHttpMessageConverter