1.创建连接：
SearchRequest request = new SearchRequest(GlobalConstant.INDEX_NAME_NEW);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
request.source(searchSourceBuilder);
RestHighLevelClient client = null;
try {
client = getNewClient();
SearchResponse search = client.search(request, RequestOptions.DEFAULT);
Aggregations aggregations = search.getAggregations();
Terms byCompanyAggregation2 = aggregations.get("by_code");
//新的ES地址
public static volatile String CLUSTER_NODES_NEW;
//新的ES地址端口
public static volatile Integer CLUSTER_NODES_NEW_PORT;
//新的ES日志索引
public static volatile String INDEX_NAME_NEW;
//报文日志索引
public static volatile String API_MSG_INDEX;
//新的ES登录账号
public static volatile String ES_USER_NAME;
//新的ES地址登录密码
public static volatile String ES_USER_PWD;
public static RestHighLevelClient getNewClient(){
HttpHost host=new HttpHost(GlobalConstant.CLUSTER_NODES_NEW, GlobalConstant.CLUSTER_NODES_NEW_PORT, HttpHost.DEFAULT_SCHEME_NAME);
RestClientBuilder builder= org.elasticsearch.client.RestClient.builder(host);
builder.setMaxRetryTimeoutMillis(300 * 1000);
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(GlobalConstant.ES_USER_NAME, GlobalConstant.ES_USER_PWD));
builder.setHttpClientConfigCallback(f -> f.setDefaultCredentialsProvider(credentialsProvider));
RestClientBuilder.RequestConfigCallback configCallback = new RestClientBuilder.RequestConfigCallback() {
@Override
public org.apache.http.client.config.RequestConfig.Builder customizeRequestConfig(org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder) {
return requestConfigBuilder
.setConnectTimeout(30000)
.setSocketTimeout(300 * 1000);//更改客户端的超时限制默认30秒现在改为5分钟
}
};
builder.setRequestConfigCallback(configCallback);
RestHighLevelClient restClient = new RestHighLevelClient( builder);
return restClient;
}
2.sql查询
2.1设置查询缓存大小：
默认时100mb下面设置成500
RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
builder.setHttpAsyncResponseConsumerFactory(
new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(500 * 1024 * 1024));
RequestOptions COMMON_OPTIONS = builder.build();
response = newClient.search(request, COMMON_OPTIONS);
2.2设置查询出来的固定字段数据，超时时间
//过滤字段
String[] excludes = {};
String[] includes = {"logData.apiId", "logData.rawRequest", "logData.rawResponse", "logData.gatewayErrorCode", "logData.backendResponseCode"};
searchSourceBuilder.fetchSource(includes, excludes);
// 限制查询出来的条数 searchSourceBuilder.query(boolQueryBuilder).size(LIMIT_SIZE);
searchSourceBuilder.timeout(TimeValue.timeValueSeconds(300));
2.3查询某个区间范围
boolQueryBuilder.must(
QueryBuilders.rangeQuery("collectTime")
.gte(beginDate.getTime())
.lte(endDate.getTime()));
2.4查询某个并的数据
boolQueryBuilder.must(QueryBuilders.matchQuery("busiLogType", "eopori"));
2.5 查询含有某个字段和不含有某个字段
一下意思时 存在logData.gatewayResponseCode数据和不存在
logData.gatewayErrorCode数据或者为空
boolQueryBuilder.must(QueryBuilders.wildcardQuery("logData.gatewayResponseCode", "*"));
boolQueryBuilder.mustNot(QueryBuilders.wildcardQuery("logData.gatewayErrorCode", "*"));
以上完整可以es的sql为
{
"size":120,
"timeout":"300s",
"query":{
"bool":{
"must":[
{
"match":{
"busiLogType":{
"query":"eopori",
"operator":"OR",
"prefix_length":0,
"max_expansions":50,
"fuzzy_transpositions":true,
"lenient":false,
"zero_terms_query":"NONE",
"auto_generate_synonyms_phrase_query":true,
"boost":1
}
}
},
{
"range":{
"collectTime":{
"from":1647142201124,
"to":1649820601124,
"include_lower":true,
"include_upper":true,
"boost":1
}
}
},
{
"match":{
"logData.apiId":{
"query":"10240",
"operator":"OR",
"prefix_length":0,
"max_expansions":50,
"fuzzy_transpositions":true,
"lenient":false,
"zero_terms_query":"NONE",
"auto_generate_synonyms_phrase_query":true,
"boost":1
}
}
},
{
"wildcard":{
"logData.gatewayResponseCode":{
"wildcard":"*",
"boost":1
}
}
}
],
"must_not":[
{
"wildcard":{
"logData.gatewayErrorCode":{
"wildcard":"*",
"boost":1
}
}
}
],
"adjust_pure_negative":true,
"boost":1
}
},
"_source":{
"includes":[
"logData.apiId",
"logData.rawRequest",
"logData.rawResponse",
"logData.gatewayErrorCode",
"logData.backendResponseCode"
],
"excludes":[
]
}
}
2.6 or查询和and包含or
should就是or mus 就是and wildcardQuery就是模糊查询
BoolQueryBuilder requestQueryBuilder = QueryBuilders.boolQuery();
requestQueryBuilder.should(QueryBuilders.wildcardQuery("logData.apiId.keyword", "*" + apiId + "*"));
requestQueryBuilder.should(QueryBuilders.wildcardQuery("logData.apiName.keyword", "*" + apiId + "*"));
boolQueryBuilder.must(requestQueryBuilder);
2.7 分页查询
from 就是从哪里开始查询
size：就是查询数据
sort 就是按照logData.timestamp时间降序排序
searchSourceBuilder
.size(size)
.from(from)
.trackTotalHits(true)
.query(
boolQueryBuilder
)
.sort("logData.timestamp", SortOrder.DESC);
2.8 分组
关键词：aggregation
按照：busiLogType分组数量查查询
别名是：by_busiLogType
统计分组的数据
AggregationBuilder by_busiLogType = AggregationBuilders.terms("by_busiLogType").field("busiLogType").size(10000);
searchSourceBuilderCount.query(
boolQueryBuilder
).aggregation(
by_busiLogType
).size(0);
3.查询结果获取数据
将其转为map获取某个字段的数据在这里插入代码片
SearchHit[] results = search.getHits().getHits();
for (SearchHit hit : results) {
Map<String, Object>
map = hit.getSourceAsMap();
Map<String, Object> maplogData = (Map<String, Object>) map.get("logData");
maps.add(maplogData);
}
将其转为字符串在转为json对象字符串
if (response.status().equals(RestStatus.OK) && !response.isTimedOut()) {
SearchHits hits = response.getHits();
List<ApiMsg> apiMsgs = new ArrayList<>();
for (SearchHit hit : hits) {
String apiMsgStr = hit.getSourceAsString();
try {
String logDataStr = JSON.parseObject(apiMsgStr).getJSONObject("logData").toJSONString();
获取分组的统计数据：
Aggregations aggregations = search.getAggregations();
Terms byCompanyAggregation = aggregations.get("by_busiLogType");
for (Terms.Bucket bucket2 : byCompanyAggregation.getBuckets()) {
if (bucket2.getKey() != null && !StringUtils.isEmpty(bucket2.getKey().toString().trim()) && "eoplog".equals(bucket2.getKey().toString())) {
//总量
total = bucket2.getDocCount();
}
}
4.引用es包
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.5.0</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>6.5.0</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>6.5.0</version>
</dependency>
5. elastic相关操作
5.1查询
GET aiolog-eop-eop-comm-eopori-orderquery-2022.04.02/_search
5.2 插入
POST aiolog-eop-eop-comm-eoplog-orderquery-2022.04.12/_doc
{}
5.3 删除
DELETE aiolog-eop-eop-comm-eopori-orderquery-2022.03.21
5.4查看模板
右键点击
aiolog-eop-eop-comm-eoplog-orderquery-2022.04.12
{}
5.5插入模板
PUT /_template/aiolog-eop-eop
{}
注意：
number_of_shards和number_of_replicas es配置的分区数量
如果只有一个设置一
5.6 es的安装可以参考
https://blog.csdn.net/happyzxs/article/details/89156068
https://www.cnblogs.com/xyddm/p/14188842.html
https://www.elastic.co/cn/downloads/past-releases/#kibana
https://www.cnblogs.com/wwjj4811/p/14700279.html
https://blog.csdn.net/qq_43676531/article/details/113095349
https://blog.csdn.net/he19970408/article/details/107359861/
https://blog.csdn.net/weixin_39887748/article/details/112481376
