近期需要对线上A服务接口进行健康度监控，即把A服务各个接口每天的性能指标进行统计并写入库表，便于对接口通过周期性数据进行全面分析及接口优化。
据调研了解，有2种解决方案可行：
1）目前服务接口请求信息已全面接入elk(Elasticsearch,Logstash,Kibana)，可以统计kibana中的visualization 功能统计，通过程序模拟visualization功能请求，获取到性能指标数据。
优点：不需要程序进行任何改动，无研发同事时间成本。
缺点：需要进行尝试，未使用此种方式统计过。
2）研发同事在程序中，使用prometheus，对各接口调用时进行埋点，采集接口请求、返回信息。
Prometheus介绍 - 运维人在路上 - 博客园
优点：prometheus采集数据日常较常用、成熟，且采集后可以进行多维度统计数据。
缺点：需要研发同事介入、增加了研发时间成本。
经过考虑，先尝试方案一（后经实践，可行）。
以下为方案一的具体实现细节，供参考。
一、在elastic中，确定好服务查询索引及查询条件。
查询索引：logstash-lb-nginx-logs-xxxx-*
查询条件：http_host: "d-es-xxxx.xxxx.com" AND request_uri: "/kw/segmentWordList"
二、使用kibana的visualization功能，画出接口99响应时间及QPS统计图。
最终展示：接口99响应时间及QPS统计图详细创建步骤请参见：elastic中创建服务接口99响应时间及QPS统计图_见贤思齐IT的博客-CSDN博客，在这就不赘述。
- 接口99分位响应时间，如下图所示 ：
- 接口QPS，如下图所示：
三、通过展示图，获取请求body信息（json)。
以下为接口qps为示例，详细介绍下步骤：
1、在dashboards中查询已创建的可视面板。
2、打开inspect
3、在 弹出的页面中，选择Request选项
4、终于看到心心念的json请求信息了。
json信息如下：
{
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"fixed_interval": "1s",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
},
"aggs": {
"1": {
"percentiles": {
"field": "request_time",
"percents": [
95,
99
],
"keyed": false
}
}
}
}
},
"size": 0,
"fields": [
{
"field": "@timestamp",
"format": "date_time"
},
{
"field": "kafka.block_timestamp",
"format": "date_time"
}
],
"script_fields": {},
"stored_fields": [
"*"
],
"runtime_mappings": {},
"_source": {
"excludes": []
},
"query": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [
{
"match_phrase": {
"http_host": "d-es-xx.zpidc.com"
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"match_phrase": {
"request_uri": "/kw/segmentWordList"
}
}
],
"minimum_should_match": 1
}
}
]
}
},
{
"range": {
"@timestamp": {
"gte": "2022-03-30T07:29:17.066Z",
"lte": "2022-03-31T07:29:17.066Z",
"format": "strict_date_optional_time"
}
}
}
],
"should": [],
"must_not": []
}
}
}
四、以下为java代码具体实现，仅供参考。
注：第三节仅是单个接口的展示，在java代码中可以获取服务下所有接口，并参数化请求信息 。
package statistical;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.zhilian.statistics.utils.HttpGetPost;
import org.jooq.DSLContext;
import org.jooq.Record;
import org.jooq.Result;
import org.jooq.SQLDialect;
import org.jooq.impl.DSL;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.StringUtils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.time.LocalDate;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import static com.zhilian.statistics.jooq.public_.Tables.*;
/**
* @Author: jiajun.du
* @Date: 2022/3/30 20:50
*/
public class StatisticsNlp {
private static Logger logger = LoggerFactory.getLogger(StatisticsNlp.class);
HttpGetPost httpGetPost = new HttpGetPost();
//连接数据库配置信息
public static String userName = "xxxx";
public static String password = "xxxx";
public static String url = "jdbc:postgresql://172.xx.xx.xx:1601/business_data_platform?tcpKeepAlive=true&autoReconnect=true";
public static Connection conn;
public static DSLContext dsl;
private static long day = 0;
public static LocalDate localDate = LocalDate.now().plusDays(day);//获取当前日期
static {
try {
conn = DriverManager.getConnection(url, userName, password); //创建连接
dsl = DSL.using(conn, SQLDialect.POSTGRES);//将连接和具体的数据库类型绑定
} catch (SQLException e) {
e.printStackTrace();
}
}
//计算服务接口最大值（99分位耗时、最大QPS）
public float maxValue(JSONArray array,String name){
float max99Value = 0f;
float param = 0f;
String valueStr = null;
if(array!=null && array.size()>=1){
for (int k = 0; k < array.size(); k++) {
if(name.equals("time_99line")){//99分位耗时统计
valueStr = array.getJSONObject(k).getJSONObject("1").getString("values");
String p = valueStr.replace("[", "").replace("]", "");
JSONObject jsonObject = JSONObject.parseObject(p);
param = Float.parseFloat(jsonObject.getString("value"));
}else if(name.equals("max_pqs")){//最大QPS统计
param = Float.parseFloat(array.getJSONObject(k).getString("doc_count"));
}
if (max99Value < param) {
max99Value = param;
}
}
}
return max99Value;
}
//更新服务字典表中接口列表信息
public Boolean nlpMethodUpdate(String index,String body)throws Exception{
JSONObject methodJsonResult = null;//存放请求结果json
JSONObject bodyJson = null;//查询body信息
JSONArray methodJsonArray = null;
String strParma = null; //临时方法名称变量
String realmName = "d-es-thirdparty.zpidc.com";
String moudle = null;
String strs [] = null;
List<String [][]> nlpMethods = new ArrayList<>();
if(url!=null){
bodyJson = JSONObject.parseObject(body);
methodJsonResult = HttpGetPost.doPost(index, bodyJson, "UTF-8");
if(methodJsonResult!=null && methodJsonResult.getJSONObject("aggregations").size()>=1){
methodJsonArray = methodJsonResult.getJSONObject("aggregations").getJSONObject("2").getJSONArray("buckets");//获取请求结果中array信息
for(int i = 0;i<methodJsonArray.size();i++){
String method [][] = new String[1][2];
strParma = methodJsonArray.getJSONObject(i).getString("key");//获取接口方法名称
if(strParma!=null && strParma.length()>1){
strParma = strParma.replace("http://"+realmName, "");
strs = strParma.split("/");
moudle = strs[1];
method[0][0] = strParma;
method[0][1] = moudle;
nlpMethods.add(i,method);
}
}
}
if(nlpMethods!=null && nlpMethods.size()>=1){
for(String [][] methodName : nlpMethods){
List<Record> r = dsl.select().from(SERVICE_NLP_DATA_DICTIONARY).where(SERVICE_NLP_DATA_DICTIONARY.METHOD_NAME.equal(methodName[0][0])).fetch();
if(r.size()==0){
dsl.insertInto(SERVICE_NLP_DATA_DICTIONARY)
.set(SERVICE_NLP_DATA_DICTIONARY.SERVICE_NAME,realmName)
.set(SERVICE_NLP_DATA_DICTIONARY.METHOD_NAME,methodName[0][0])
.set(SERVICE_NLP_DATA_DICTIONARY.REMARK,"update")
.set(SERVICE_NLP_DATA_DICTIONARY.MODULE,moudle)
.returning(SERVICE_NLP_DATA_DICTIONARY.ID)
.fetchOne();
}
}
}
}
return null;
}
//统计nlp服务接口99分位耗时、maxqps性能指标
public Boolean nlpDataSource(String index,String [][] querys,Record r) throws Exception {
JSONObject jsonObjectTimeBody = null;
JSONObject jsonObjectQpsBody =null;
JSONObject jsonObjectTimeResponse = null;
JSONObject jsonObjectQpsResponse = null;
float time_99 = 0f;//99分位耗时
float max_qps =0f;
if(querys!=null&&querys.length>=1){
JSONArray jsonArrayTime = null;
JSONArray jsonArrayQps = null;
for(int i=0;i< querys.length;i++) {
if (querys[i][0].equals("time_99line")) {
jsonObjectTimeBody = JSON.parseObject(querys[i][1]);
jsonObjectTimeResponse = HttpGetPost.doPost(index, jsonObjectTimeBody, "UTF-8");
} else if (querys[i][0].equals("max_pqs")) {
jsonObjectQpsBody = JSON.parseObject(querys[i][1]);
jsonObjectQpsResponse = HttpGetPost.doPost(index, jsonObjectQpsBody, "UTF-8");
}
}
if(jsonObjectTimeResponse!=null && jsonObjectTimeResponse.getJSONObject("aggregations").size()>=1){
jsonArrayTime = jsonObjectTimeResponse.getJSONObject("aggregations").getJSONObject("2").getJSONArray("buckets");
time_99 = maxValue(jsonArrayTime,"time_99line");
}
if(jsonObjectQpsResponse!=null && jsonObjectQpsResponse.getJSONObject("aggregations").size()>=1){
jsonArrayQps = jsonObjectQpsResponse.getJSONObject("aggregations").getJSONObject("2").getJSONArray("buckets");
max_qps = maxValue(jsonArrayQps,"max_pqs");
}
//服务接口性能指标信息入库
dsl.insertInto(SERVICE_NLP_INTERFACE_INDICATORS)
.set(SERVICE_NLP_INTERFACE_INDICATORS.SERVICE_NAME,r.getValue("service_name").toString())
.set(SERVICE_NLP_INTERFACE_INDICATORS.MODULE,r.getValue("module").toString())
.set(SERVICE_NLP_INTERFACE_INDICATORS.METHOD_NAME,r.getValue("method_name").toString())
.set(SERVICE_NLP_INTERFACE_INDICATORS.MAX_99_LINE,time_99)
.set(SERVICE_NLP_INTERFACE_INDICATORS.MAX_QPS,max_qps)
.set(SERVICE_NLP_INTERFACE_INDICATORS.DATE,localDate.plusDays(day))
.returning(SERVICE_NLP_INTERFACE_INDICATORS.ID)
.fetchOne();
}
return null;
}
public static void main(String[] args) throws Exception {
StatisticsNlp statisticsNlp = new StatisticsNlp();
String index = "http://ops-xx-xx.xx.com/logstash-lb-nginx-logs-xxxx-*/_search";//查询索引;//接口、渠道级别请求地址前缀
//统计线上A服务下所有接口信息（body体信息）
String bodyStr = "{\"aggs\":{\"2\":{\"terms\":{\"field\":\"url_path.keyword\",\"order\":{\"_count\":\"desc\"},\"size\":100}}},\"size\":0,\"fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"kafka.block_timestamp\",\"format\":\"date_time\"}],\"script_fields\":{},\"stored_fields\":[\"*\"],\"runtime_mappings\":{},\"_source\":{\"excludes\":[]},\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"bool\":{\"should\":[{\"match_phrase\":{\"http_host\":\"d-es-xxxx.xxxx.com\"}}],\"minimum_should_match\":1}},{\"range\":{\"@timestamp\":{\"gte\":\""+localDate.plusDays(day-3)+"T00:00:00.000Z\",\"lte\":\""+localDate.plusDays(day)+"T23:59:00.000Z\",\"format\":\"strict_date_optional_time\"}}}],\"should\":[],\"must_not\":[]}}}";
statisticsNlp.nlpMethodUpdate(index,bodyStr);//更新服务接口列表
Result<Record> nlp_result = dsl.select().from(SERVICE_NLP_DATA_DICTIONARY).fetch();//获取服务列表
String [][] querys = new String[2][2];//存放请求信息
for(int t = 0 ; t < 1; t++){
day = t * -1;
for(Record r:nlp_result){//遍历nlp服务接口
//time_99line
querys[0][0] = "time_99line";
querys[0][1] = "{\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"fixed_interval\":\"1m\",\"time_zone\":\"Asia/Shanghai\",\"min_doc_count\":1},\"aggs\":{\"1\":{\"percentiles\":{\"field\":\"request_time\",\"percents\":[99],\"keyed\":false}}}}},\"size\":0,\"fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"kafka.block_timestamp\",\"format\":\"date_time\"}],\"script_fields\":{},\"stored_fields\":[\"*\"],\"runtime_mappings\":{},\"_source\":{\"excludes\":[]},\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"match_phrase\":{\"http_host\":\"d-es-xxxx.xxxx.com\"}}],\"minimum_should_match\":1}},{\"bool\":{\"should\":[{\"match_phrase\":{\"request_uri\":\""+r.getValue("method_name")+"\"}}],\"minimum_should_match\":1}}]}},{\"range\":{\"@timestamp\":{\"gte\":\""+localDate.plusDays(day)+"T08:00:00.066Z\",\"lte\":\""+localDate.plusDays(day)+"T23:00:17.066Z\",\"format\":\"strict_date_optional_time\"}}}],\"should\":[],\"must_not\":[]}}}";
//maxqps
querys[1][0] = "max_pqs";
querys[1][1] = "{\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"fixed_interval\":\"1s\",\"time_zone\":\"Asia/Shanghai\",\"min_doc_count\":1}}},\"size\":0,\"fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"kafka.block_timestamp\",\"format\":\"date_time\"}],\"script_fields\":{},\"stored_fields\":[\"*\"],\"runtime_mappings\":{},\"_source\":{\"excludes\":[]},\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"match_phrase\":{\"http_host\":\"d-es-xxxx.xxxx.com\"}}],\"minimum_should_match\":1}},{\"bool\":{\"should\":[{\"match_phrase\":{\"request_uri\":\""+r.getValue("method_name")+"\"}}],\"minimum_should_match\":1}}]}},{\"range\":{\"@timestamp\":{\"gte\":\""+localDate.plusDays(day)+"T08:00:09.025Z\",\"lte\":\""+localDate.plusDays(day)+"T23:00:12.479Z\",\"format\":\"strict_date_optional_time\"}}}],\"should\":[],\"must_not\":[]}}}";
statisticsNlp.nlpDataSource(index,querys,r);
}
}
}
}
package com.zhilian.statistics.utils;
import com.alibaba.fastjson.JSONObject;
import org.apache.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;a
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.text.ParseException;
/**
* @Author: jiajun.du
* @Date: 2021/11/23 15:53
*/
public class HttpGetPost {
Logger logger = LoggerFactory.getLogger(HttpGetPost.class);
/**
* get请求
* @param url
* @return
* @throws Exception
*/
public JSONObject doGet(String url, String param) throws Exception{
CloseableHttpClient httpClient = null;
CloseableHttpResponse httpResponse =null;
String strResult = "";
try {
httpClient = HttpClients.createDefault();//创建一个httpClient实例
org.apache.http.client.methods.HttpGet httpGet =new org.apache.http.client.methods.HttpGet(url+param);//创建httpGet远程连接实例
// 设置请求头信息
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(300000).//连接主机服务器时间
setConnectionRequestTimeout(300000).//请求超时时间
setSocketTimeout(300000).//数据读取超时时间
build();
httpGet.setConfig(requestConfig);//为httpGet实例设置配置信息
httpResponse = httpClient.execute(httpGet);//通过get请求得到返回对象
//发送请求成功并得到响应
if(httpResponse.getStatusLine().getStatusCode()==200){
strResult = EntityUtils.toString(httpResponse.getEntity());
JSONObject resultJsonObject = JSONObject.parseObject(strResult);//获取请求返回结果
return resultJsonObject;
}else{
logger.error("请求{}失败,\n状态码为{}",url,httpResponse.getStatusLine().getStatusCode());
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException i){
i.printStackTrace();
logger.error("IOException异常：{}",i.getMessage());
} finally {
if(httpResponse!=null){
try {
httpResponse.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(httpClient!=null){
try {
httpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return null;
}
/**
* 发送post请求
* @param url 路径
* @param jsonObject 参数(json类型)
* @param encoding 编码格式
* @return
* @throws ParseException
* @throws IOException
*/
public static JSONObject doPost(String url, JSONObject jsonObject,String encoding) throws ParseException, IOException{
JSONObject object = null;
String body = "";
//创建httpclient对象
CloseableHttpClient client = HttpClients.createDefault();
//创建post方式请求对象
HttpPost httpPost = new HttpPost(url);
//装填参数
StringEntity s = new StringEntity(jsonObject.toString(), "utf-8");
//s.setContentEncoding(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
//设置参数到请求对象中
httpPost.setEntity(s);
System.out.println("请求地址："+url);
//设置header信息
//指定报文头【Content-type】、【User-Agent】
httpPost.setHeader("Content-Type", "application/json");
httpPost.setHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");
httpPost.setHeader("Connection","keep-alive");
//执行请求操作，并拿到结果（同步阻塞）
CloseableHttpResponse response = client.execute(httpPost);
//获取结果实体
HttpEntity entity = response.getEntity();
if (entity != null) {
//按指定编码转换结果实体为String类型
body = EntityUtils.toString(entity, encoding);
object = JSONObject.parseObject(body);
}
EntityUtils.consume(entity);
//释放链接
response.close();
return object;
}
}
落库结果一览（隐私信息已模糊处理）：
版权归原作者 ~见贤思齐~ 所有， 如有侵权，请联系我们删除。