0


高并发高可用之ElasticSearch

目录

ES里面的数据怎么保持与mysql实时同步?
都存内存 数据不会越来越多吗?有过期时间吗?

ES对比MySql数据库

ES的数据存储在磁盘中,数据操作在内存中。

  • 索引:数据库
  • 类型:数据表
  • 文档:表里的数据
  • 属性:表列名

在这里插入图片描述
注意:ElasticSearch6.0之后移除了类型的概念。7.x使用类型会警告,8.x将彻底废除。

Docker下安装ES和kibana

安装ES

# 将docker里的目录挂载到linux的/mydata目录中# 修改/mydata就可以改掉docker里的mkdir -p /mydata/elasticsearch/config
mkdir -p /mydata/elasticsearch/data

# es可以被远程任何机器访问echo"http.host: 0.0.0.0">/mydata/elasticsearch/config/elasticsearch.yml

# 递归更改文件访问权限,es需要访问chmod -R 777 /mydata/elasticsearch/
docker pull elasticsearch:7.4.2
docker pull kibana:7.4.2
版本要统一
# 9200是用户交互端口 9300是集群心跳端口# -e指定是单阶段运行# -e ES_JAVA_OPTS="-Xms64m -Xmx512m"指定初始占用内存大小和最大占用大小# 反斜杠表示换行docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node"\
-e ES_JAVA_OPTS="-Xms64m -Xmx512m"\
-v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-d elasticsearch:7.4.2

# 设置开机启动elasticsearchdocker update elasticsearch --restart=always

查看日志命令:

docker logs elasticsearch

查看docker镜像ID命令:

docker ps -a

运行docker镜像:

docker start 镜像ID

访问:
在这里插入图片描述

安装kibana

# 指定了ES交互端口9200和IP地址docker run --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.239.134:9200 -p 5601:5601 -d kibana:7.4.2

# 设置开机启动kibanadocker update kibana  --restart=always

kibana访问地址:http://192.168.239.134:5601/

增删改查操作

(1)GET /_cat/nodes:查看所有节点
(2)GET /_cat/health:查看es健康状况
(3)GET /_cat/master:查看主节点
(4)GET /_cat/indices:查看所有索引 ,等价于mysql数据库的show databases;

新增/更新
PUT/POST /索引名/类型名/ID

http://192.168.56.10:9200/索引名/类型名/ID
请求参数Json:
{"name":"John Doe"}

支持put和post,post不写ID可以自动生产。对一个ID多次操作都会变为update操作。

查询
GET /索引名/类型名/ID

更新

POST /索引名/类型名/ID/_update
{"doc":{"name":"111"}}

加_update参数就要加doc。
POST时带_update会对比元数据,如果一样就不进行任何操作。

删除
删除文档数据
DELETE /索引名/类型名/ID
删除索引
DELETE /索引名

注:elasticsearch并没有提供删除类型的操作,只提供了删除索引和文档的操作。

批量执行
在指定索引和类型下批量执行
POST /索引名/类型名/_bulk
在整个ES中批量执行
POST /_bulk

高级检索Query DSL

  1. query/match匹配查询 如果是非字符串,会进行精确匹配。如果是字符串,会进行全文检索GET bank/_search{"query":{"match":{"account_number":"20"}}}
  2. query/match_phrase 【不拆分匹配】 将需要匹配的值当成一整个单词(不分词)进行检索。 – match_phrase:不拆分字符串进行检索,包含就匹配成功。 – 字段.keyword:必须全匹配上才检索成功。GET bank/_search{"query":{"match_phrase":{"address":"990 Mill"}}}``````GET bank/_search{"query":{"match":{"address.keyword":"990 Mill" # 字段后面加上 .keyword }}}
  3. query/multi_math 【多字段匹配】GET bank/_search{"query":{"multi_match":{ # 前面的match仅指定了一个字段。 "query":"mill","fields":[ # state和address有mill子串 不要求都有 "state","address"]}}}
  4. query/bool/must 【复合查询】 – must:必须达到must所列举的所有条件 – must_not:必须不匹配must_not所列举的所有条件。 – should:应该满足should所列举的条件。满足条件最好,不满足也可以,满足得分更高GET bank/_search{"query":{"bool":{"must":[{"match":{"gender":"M"}},{"match":{"address":"mill"}}],"must_not":[{"match":{"age":"18"}}],"should":[{"match":{"lastname":"Wallace"}}]}}}
  5. query/filter 【结果过滤】 must 贡献得分 should 贡献得分 must_not 不贡献得分 filter 不贡献得分GET bank/_search{"query":{"bool":{"must":[{"match":{"address":"mill"}}],"filter":{ # query.bool.filter "range":{"balance":{ # 哪个字段 "gte":"10000","lte":"20000"}}}}}}
  6. query/term 和match一样。匹配某个属性的值。 – 全文检索字段用match, – 其他非text文本字段匹配用term。
  7. aggs 【聚合】 复杂子聚合例子:查出所有年龄分布,并且这些年龄段中M的平均薪资和F的平均薪资以及这个年龄段的总体平均薪资GET bank/_search{"query":{"match_all":{}},"aggs":{"ageAgg":{"terms":{ # 看age分布 "field":"age","size":100},"aggs":{ # 子聚合 "genderAgg":{"terms":{ # 看gender分布 "field":"gender.keyword" # 注意这里,文本字段应该用.keyword },"aggs":{ # 子聚合 "balanceAvg":{"avg":{ # 男性的平均 "field":"balance"}}}},"ageBalanceAvg":{"avg":{ #age分布的平均(男女) "field":"balance"}}}}},"size":0}

更多Aggregations聚合函数请参考官方文档:https://www.elastic.co/guide/en/elasticsearch/reference/7.5/search-aggregations.html

映射

存入数据后ES会把字段自动映射一个数据类型。如果自动映射的数据类型不正确还可以手动指定映射。
创建索引并指定映射

PUT /my_index
{"mappings":{"properties":{"age":{"type":"integer"},"email":{"type":"keyword" # 指定为keyword
      },"name":{"type":"text" # 全文检索。保存时候分词,检索时候进行分词匹配
      }}}}

查看映射:GET /my_index

有映射的情况下添加新的字段并指定映射

PUT /my_index/_mapping
{"properties":{"employee-id":{"type":"keyword","index":false # 字段不能被检索。检索
    }}}

更新映射
由于改变映射会影响到该字段下的数据,故想要更新映射只支持把数据迁移到新的映射规则下。
数据迁移:

POST _reindex
{"source":{"index":"bank",        #数据源索引
    "type":"account"         #6.0后没有类型可以不写该行
  },"dest":{"index":"newbank"        #要迁移到的新索引
  }}

安装中文IK分词器

下载并解压elasticsearch-analysis-ik-7.4.2到安装ES时挂载的插件外部目录/mydata/elasticsearch/plugins
配置ik插件目录访问权限并重启ES容器
注意:IK版本必须和ES版本一致

使用
支持两种分词模式:ik_smart , ik_max_word

GET _analyze
{"analyzer":"ik_smart","text":"我是中国人"}

扩展IK分词器有两种方式

  1. 编写一个项目让IK访问。
  2. 词条配置到一个nginx让IK访问 – 在nginx的html目录下创建es目录并创建fenci.txt文件,在fenci.txt中写入自定义的词语,每行一条。 – 修改/plugins/ik/config中的IKAnalyzer.cfg.xml文件: – 配置远程扩展字典访问地址<?xml version="1.0"encoding="UTF-8"?><!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"><properties><comment>IK Analyzer 扩展配置</comment><!--用户可以在这里配置自己的扩展字典 --><entry key="ext_dict"></entry><!--用户可以在这里配置自己的扩展停止词字典--><entry key="ext_stopwords"></entry><!--用户可以在这里配置远程扩展字典 --><entry key="remote_ext_dict">http://192.168.56.10/es/fenci.txt</entry><!--用户可以在这里配置远程扩展停止词字典--><!-- <entry key="remote_ext_stopwords">words_location</entry> --></properties>

参考:https://github.com/medcl/elasticsearch-analysis-ik

SpringBoot整合ES

推荐使用Elasticsearch-Rest-Client:官方RestClient,封装了ES操作,API层次分明,上手简单。

  1. 创建一个es-search微服务,可以勾选spring web组件,依赖common模块,配置注册中心,配置中心等配置
  2. 引入maven依赖,依赖版本要和ES版本保持一致<dependency><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-high-level-client</artifactId><version>7.4.2</version></dependency>由于当前spring-boot版本默认依赖管理的ES版本是6.8.5,故要改为手动管理ES版本<properties><java.version>1.8</java.version><elasticsearch.version>7.4.2</elasticsearch.version></properties>
  3. 编写ES配置类@ConfigurationpublicclassESConfig{//对所有请求进行配置项publicstaticfinalRequestOptions COMMON_OPTIONS;static{RequestOptions.Builder builder =RequestOptions.DEFAULT.toBuilder(); COMMON_OPTIONS = builder.build();}@BeanpublicRestHighLevelClientesRestClient(){// 这里可以一次性指定多个esRestClientBuilder builder =RestClient.builder(newHttpHost("192.168.239.134",9200,"http"));RestHighLevelClient client =newRestHighLevelClient(builder);return client;}}
  4. 使用,参考官方文档 https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-getting-started-initialization.html```packagecom.example.essearch;importcom.alibaba.fastjson.JSON;importcom.example.essearch.config.ESConfig;importorg.elasticsearch.action.index.IndexRequest;importorg.elasticsearch.action.index.IndexResponse;importorg.elasticsearch.action.search.SearchRequest;importorg.elasticsearch.action.search.SearchResponse;importorg.elasticsearch.client.RestHighLevelClient;importorg.elasticsearch.common.xcontent.XContentType;importorg.elasticsearch.index.query.QueryBuilders;importorg.elasticsearch.search.SearchHit;importorg.elasticsearch.search.SearchHits;importorg.elasticsearch.search.aggregations.AggregationBuilders;importorg.elasticsearch.search.aggregations.Aggregations;importorg.elasticsearch.search.aggregations.bucket.terms.Terms;importorg.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder;importorg.elasticsearch.search.aggregations.metrics.Avg;importorg.elasticsearch.search.aggregations.metrics.AvgAggregationBuilder;importorg.elasticsearch.search.builder.SearchSourceBuilder;importorg.junit.jupiter.api.Test;importorg.springframework.beans.factory.annotation.Autowired;importorg.springframework.boot.test.context.SpringBootTest;importjava.io.IOException;@SpringBootTestclassEsSearchApplicationTests{@AutowiredprivateRestHighLevelClient client;/** * 创建/更新索引 * @throws IOException */@TestpublicvoidindexData()throwsIOException{User user =newUser(); user.setUserName("张三"); user.setAge(20); user.setGender("男");String jsonString = JSON.toJSONString(user);// 设置索引,索引名为usersIndexRequest indexRequest =newIndexRequest("users"); indexRequest.id("1");//设置要保存的内容,指定数据和类型 indexRequest.source(jsonString,XContentType.JSON);//执行创建索引和保存数据IndexResponse index = client.index(indexRequest,ESConfig.COMMON_OPTIONS);System.out.println(index);}/** * 高级检索与聚合分析 * @throws IOException */@TestpublicvoidsearchData()throwsIOException{SearchSourceBuilder sourceBuilder =newSearchSourceBuilder();// 构造检索条件//sourceBuilder.query();//sourceBuilder.from();//sourceBuilder.size();//sourceBuilder.aggregation(); sourceBuilder.query(QueryBuilders.matchQuery("address","mill"));// 聚合//AggregationBuilders工具类构建AggregationBuilder// 构建第一个聚合条件:按照年龄的值分布TermsAggregationBuilder agg1 =AggregationBuilders.terms("agg1").field("age").size(10);// 设置聚合名称为agg1 sourceBuilder.aggregation(agg1);// 构建第二个聚合条件:平均薪资AvgAggregationBuilder agg2 =AggregationBuilders.avg("agg2").field("balance");// 设置聚合名称为agg2 sourceBuilder.aggregation(agg2);System.out.println("检索条件"+sourceBuilder.toString());// 1 创建检索请求SearchRequest searchRequest =newSearchRequest(); searchRequest.indices("bank");//设置请求索引为bank searchRequest.source(sourceBuilder);// 2 执行检索SearchResponse response = client.search(searchRequest,ESConfig.COMMON_OPTIONS);// 3 分析响应结果System.out.println(response.toString());// 3.1 获取java beanSearchHits hits = response.getHits();SearchHit[] hitsList = hits.getHits();for(SearchHit hit : hitsList){ hit.getId(); hit.getIndex();String sourceAsString = hit.getSourceAsString();Account account = JSON.parseObject(sourceAsString,Account.class);System.out.println(account);}// 3.2 获取检索到的聚合分析信息Aggregations aggregations = response.getAggregations();Terms agg1Terms = aggregations.get("agg1");for(Terms.Bucket bucket : agg1Terms.getBuckets()){String keyAsString = bucket.getKeyAsString();System.out.println("年龄:"+keyAsString+"=====>"+bucket.getDocCount());}Avg agg2Avg = aggregations.get("agg2");System.out.println("平均薪资:"+agg2Avg.getValue());}classUser{privateString userName;privateInteger age;privateString gender;publicStringgetUserName(){return userName;}publicvoidsetUserName(String userName){this.userName = userName;}publicIntegergetAge(){return age;}publicvoidsetAge(Integer age){this.age = age;}publicStringgetGender(){return gender;}publicvoidsetGender(String gender){this.gender = gender;}}staticclassAccount{privateint account_number;privateint balance;privateString firstname;privateString lastname;privateint age;privateString gender;privateString address;privateString employer;privateString email;privateString city;privateString state;publicvoidsetAccount_number(int account_number){this.account_number = account_number;}publicintgetAccount_number(){returnthis.account_number;}publicvoidsetBalance(int balance){this.balance = balance;}publicintgetBalance(){returnthis.balance;}publicvoidsetFirstname(String firstname){this.firstname = firstname;}publicStringgetFirstname(){returnthis.firstname;}publicvoidsetLastname(String lastname){this.lastname = lastname;}publicStringgetLastname(){returnthis.lastname;}publicvoidsetAge(int age){this.age = age;}publicintgetAge(){returnthis.age;}publicvoidsetGender(String gender){this.gender = gender;}publicStringgetGender(){returnthis.gender;}publicvoidsetAddress(String address){this.address = address;}publicStringgetAddress(){returnthis.address;}publicvoidsetEmployer(String employer){this.employer = employer;}publicStringgetEmployer(){returnthis.employer;}publicvoidsetEmail(String email){this.email = email;}publicStringgetEmail(){returnthis.email;}publicvoidsetCity(String city){this.city = city;}publicStringgetCity(){returnthis.city;}publicvoidsetState(String state){this.state = state;}publicStringgetState(){returnthis.state;}}}```

实战应用

ES数据模型结构的设计
空间和时间不可兼得两种只能选其一。
方案1:

{
    skuId:1
    spuId:11
    skyTitile:华为xx
    price:999
    saleCount:99
    attr:[{尺寸:5},{CPU:高通945},{分辨率:全高清}]
缺点:如果每个sku都存储规格参数(如尺寸),会有冗余存储,因为每个spu对应的sku的规格参数都一样

方案2:

sku索引
{
    spuId:1
    skuId:11}
attr索引
{
    skuId:11
    attr:[{尺寸:5},{CPU:高通945},{分辨率:全高清}]}
先找到4000个符合要求的spu,再根据4000个spu查询对应的属性,封装了4000个id,
每次传输大小:如id为long类型,8B*4000=32000B=32KB
1K个人检索,就是32MB,高并发下会造成严重阻塞。

结论:如果将规格参数单独建立索引,会出现检索时出现大量数据传输的问题,会引起网络网络

创建索引并设置映射

PUT product
{"mappings":{"properties":{"skuId":{"type":"long"},"spuId":{"type":"keyword"},  # 不可分词
            "skuTitle":{"type":"text","analyzer":"ik_smart"  # 中文分词器
            },"skuPrice":{"type":"keyword"},"skuImg":{"type":"keyword","index":false,  # 降低占用空间,不可被检索,不生成索引,只用做页面展示
                "doc_values":false # 降低占用空间,不可被聚合,默认为true},"saleCount":{"type":"long"},"hasStock":{"type":"boolean"},"hotScore":{"type":"long"},"brandId":{"type":"long"},"catalogId":{"type":"long"},"brandName":{"type":"keyword"},"brandImg":{"type":"keyword","index":false,"doc_values":false},"catalogName":{"type":"keyword"},"attrs":{"type":"nested",    # 重要!!!表示嵌入式,防止被ES自动扁平化处理
                "properties":{"attrId":{"type":"long"},"attrName":{"type":"keyword","index":false,"doc_values":false},"attrValue":{"type":"keyword"}}}}}}

创建ES数据模型实体类

@DatapublicclassSkuEsModel{privateLong skuId;privateLong spuId;privateString skuTitle;privateBigDecimal skuPrice;privateString skuImg;privateLong saleCount;privateBoolean hasStock;privateLong hotScore;privateLong brandId;privateLong catalogId;privateString brandName;privateString brandImg;privateString catalogName;privateList<Attr> attrs;@DatapublicstaticclassAttr{privateLong attrId;privateString attrName;privateString attrValue;}}

封装数据到ES数据模型实体类并存入ES
商品上架的同时进行封装商品数据并远程调用ES微服务保存到ES中
(封装代码略)

编写ES微服务保存数据的Controller层

/*** 上架商品*/@PostMapping("/product")// ElasticSaveControllerpublicRproductStatusUp(@RequestBodyList<SkuEsModel> skuEsModels){boolean status;try{
        status = productSaveService.productStatusUp(skuEsModels);}catch(IOException e){
        log.error("ElasticSaveController商品上架错误: {}", e);returnR.error(BizCodeEnum.PRODUCT_UP_EXCEPTION.getCode(),BizCodeEnum.PRODUCT_UP_EXCEPTION.getMsg());}if(!status){returnR.ok();}returnR.error(BizCodeEnum.PRODUCT_UP_EXCEPTION.getCode(),BizCodeEnum.PRODUCT_UP_EXCEPTION.getMsg());}

编写ES微服务保存数据的Service层

publicclassProductSaveServiceImplimplementsProductSaveService{@ResourceprivateRestHighLevelClient client;/**
     * 将数据保存到ES
     * 用bulk代替index,进行批量保存
     * BulkRequest bulkRequest, RequestOptions options
     */@Override// ProductSaveServiceImplpublicbooleanproductStatusUp(List<SkuEsModel> skuEsModels)throwsIOException{// 1.给ES建立一个索引 productBulkRequest bulkRequest =newBulkRequest();// 2.构造保存请求for(SkuEsModel esModel : skuEsModels){// 设置es索引IndexRequest indexRequest =newIndexRequest(EsConstant.PRODUCT_INDEX);// 设置索引id
            indexRequest.id(esModel.getSkuId().toString());// json格式String jsonString = JSON.toJSONString(esModel);
            indexRequest.source(jsonString,XContentType.JSON);// 添加到文档
            bulkRequest.add(indexRequest);}// bulk批量保存BulkResponse bulk = client.bulk(bulkRequest,GuliESConfig.COMMON_OPTIONS);// TODO 是否拥有错误boolean hasFailures = bulk.hasFailures();if(hasFailures){List<String> collect =Arrays.stream(bulk.getItems()).map(item -> item.getId()).collect(Collectors.toList());
            log.error("商品上架错误:{}",collect);}return hasFailures;}}

检索查询参数模型分析
可能用到的参数:
全文检索:skuTitle->keyword
排序:saleCount(销量)、hotScore(热度分)、skuPrice(价格)
过滤:hasStock、skuPrice区间、brandId、catalog3Id、attrs(规格属性)
聚合:attrs

/**
封装页面所有可能传递过来的关键字
 * catalog3Id=225&keyword=华为&sort=saleCount_asc&hasStock=0/1&brandId=25&brandId=30
 */@DatapublicclassSearchParam{// 页面传递过来的全文匹配关键字privateString keyword;/** 三级分类id*/privateLong catalog3Id;//排序条件:sort=price/salecount/hotscore_desc/ascprivateString sort;// 仅显示有货privateInteger hasStock;/*** 价格区间 */privateString skuPrice;/*** 品牌id 可以多选 */privateList<Long> brandId;/*** 按照属性进行筛选 */privateList<String> attrs;/*** 页码*/privateInteger pageNum =1;/*** 原生所有查询属性*/privateString _queryString;}

检索返回结果模型分析

/**
 * <p>Title: SearchResponse</p>
 * Description:包含页面需要的所有信息
 */@DatapublicclassSearchResult{/** * 查询到的所有商品信息(即前面的ES数据模型实体类)*/privateList<SkuEsModel> products;/*** 当前页码*/privateInteger pageNum;/** 总记录数*/privateLong total;/** * 总页码*/privateInteger totalPages;/** 当前查询到的结果, 所有涉及到的品牌*/privateList<BrandVo> brands;/*** 当前查询到的结果, 所有涉及到的分类*/privateList<CatalogVo> catalogs;/** * 当前查询的结果 所有涉及到所有属性*/privateList<AttrVo> attrs;/** 导航页   页码遍历结果集(分页)  */privateList<Integer> pageNavs;//    ================以上是返回给页面的所有信息================/** 导航数据*/privateList<NavVo> navs =newArrayList<>();/** 便于判断当前id是否被使用*/privateList<Long> attrIds =newArrayList<>();@DatapublicstaticclassNavVo{privateString name;privateString navValue;privateString link;}@DatapublicstaticclassBrandVo{privateLong brandId;privateString brandName;privateString brandImg;}@DatapublicstaticclassCatalogVo{privateLong catalogId;privateString catalogName;}@DatapublicstaticclassAttrVo{privateLong attrId;privateString attrName;privateList<String> attrValue;}}

写出DSL检索语句,(如果是嵌入式的映射属性字段,检索查询,聚合,分析等都应该用相应的嵌入式语法nested)

GET gulimall_product/_search
{"query":{"bool":{"must":[{"match":{"skuTitle":"华为"}}], # 检索出华为
      "filter":[ # 过滤
        {"term":{"catalogId":"225"}},{"terms":{"brandId":["2"]}},{"term":{"hasStock":"false"}},{"range":{"skuPrice":{ # 价格1K~7K
              "gte":1000,"lte":7000}}},{"nested":{"path":"attrs", # 聚合名字
            "query":{"bool":{"must":[{"term":{"attrs.attrId":{"value":"6"}}}]}}}}]}},"sort":[{"skuPrice":{"order":"desc"}}],"from":0,"size":5,"highlight":{"fields":{"skuTitle":{}}, # 高亮的字段
    "pre_tags":"<b style='color:red'>",  # 前缀
    "post_tags":"</b>"},"aggs":{ # 查完后聚合
    "brandAgg":{"terms":{"field":"brandId","size":10},"aggs":{ # 子聚合
        "brandNameAgg":{  # 每个商品id的品牌
          "terms":{"field":"brandName","size":10}},"brandImgAgg":{"terms":{"field":"brandImg","size":10}}}},"catalogAgg":{"terms":{"field":"catalogId","size":10},"aggs":{"catalogNameAgg":{"terms":{"field":"catalogName","size":10}}}},"attrs":{"nested":{"path":"attrs"},"aggs":{"attrIdAgg":{"terms":{"field":"attrs.attrId","size":10},"aggs":{"attrNameAgg":{"terms":{"field":"attrs.attrName","size":10}}}}}}}}

检索查询代码实现
controller

@GetMapping(value ={"/search.html","/"})publicStringgetSearchPage(SearchParam searchParam,// 检索参数,Model model,HttpServletRequest request){
    searchParam.set_queryString(request.getQueryString());//_queryString是个字段SearchResult result=searchService.getSearchResult(searchParam);
    model.addAttribute("result", result);return"search";}

service

@Slf4j@ServicepublicclassProductSearchServiceImpl{@ResourceprivateRestHighLevelClient restHighLevelClient;/**
     *  根据请求参数检索ES数据,并将检索结果封装为系统返回响应实体类
     * @param searchParam
     * @return
     */publicSearchResultgetSearchResult(SearchParam searchParam){//根据带来的请求内容封装SearchResult searchResult=null;// 通过请求参数构建es查询请求SearchRequest request =bulidSearchRequest(searchParam);try{SearchResponse searchResponse = restHighLevelClient.search(request,ESConfig.COMMON_OPTIONS);// 将es响应数据封装成结果
            searchResult =bulidSearchResult(searchParam,searchResponse);}catch(IOException e){
            e.printStackTrace();}return searchResult;}/**
     * 通过请求参数构建ES查询请求
     * @param searchParam
     * @return
     */privateSearchRequestbulidSearchRequest(SearchParam searchParam){// 用于构建DSL语句SearchSourceBuilder searchSourceBuilder =newSearchSourceBuilder();//1. 构建bool queryBoolQueryBuilder boolQueryBuilder =newBoolQueryBuilder();//1.1 bool mustif(!StringUtils.isEmpty(searchParam.getKeyword())){
            boolQueryBuilder.must(QueryBuilders.matchQuery("skuTitle", searchParam.getKeyword()));}//使用不参与评分的filter,性能效率更高//1.2 bool filter//1.2.1 catalogif(searchParam.getCatalog3Id()!=null){
            boolQueryBuilder.filter(QueryBuilders.termQuery("catalogId", searchParam.getCatalog3Id()));}//1.2.2 brandif(searchParam.getBrandId()!=null&&searchParam.getBrandId().size()>0){//值有多个为List时termsQuery
            boolQueryBuilder.filter(QueryBuilders.termsQuery("brandId",searchParam.getBrandId()));}//1.2.3 hasStockif(searchParam.getHasStock()!=null){
            boolQueryBuilder.filter(QueryBuilders.termQuery("hasStock", searchParam.getHasStock()==1));}//1.2.4 priceRange//解析自定义的区间参数格式,这里为0_6000,_6000,6000_分别表示大于0小于6000,小于6000,大于6000RangeQueryBuilder rangeQueryBuilder =QueryBuilders.rangeQuery("skuPrice");if(!StringUtils.isEmpty(searchParam.getSkuPrice())){String[] prices = searchParam.getSkuPrice().split("_");if(prices.length ==1){if(searchParam.getSkuPrice().startsWith("_")){
                    rangeQueryBuilder.lte(Integer.parseInt(prices[0]));}else{
                    rangeQueryBuilder.gte(Integer.parseInt(prices[0]));}}elseif(prices.length ==2){//_6000会截取成["","6000"]if(!prices[0].isEmpty()){
                    rangeQueryBuilder.gte(Integer.parseInt(prices[0]));}
                rangeQueryBuilder.lte(Integer.parseInt(prices[1]));}
            boolQueryBuilder.filter(rangeQueryBuilder);}//1.2.5 attrs-nested  嵌入式属性使用嵌入式语法//attrs=1_5寸:8寸&2_16G:8GList<String> attrs = searchParam.getAttrs();BoolQueryBuilder queryBuilder =newBoolQueryBuilder();if(attrs!=null&&attrs.size()>0){
            attrs.forEach(attr->{String[] attrSplit = attr.split("_");
                queryBuilder.must(QueryBuilders.termQuery("attrs.attrId", attrSplit[0]));String[] attrValues = attrSplit[1].split(":");
                queryBuilder.must(QueryBuilders.termsQuery("attrs.attrValue", attrValues));});}NestedQueryBuilder nestedQueryBuilder =QueryBuilders.nestedQuery("attrs", queryBuilder,ScoreMode.None);
        boolQueryBuilder.filter(nestedQueryBuilder);//1.X bool query构建完成
        searchSourceBuilder.query(boolQueryBuilder);//2. sort  eg:sort=saleCount_desc/ascif(!StringUtils.isEmpty(searchParam.getSort())){String[] sortSplit = searchParam.getSort().split("_");
            searchSourceBuilder.sort(sortSplit[0], sortSplit[1].equalsIgnoreCase("asc")?SortOrder.ASC :SortOrder.DESC);}//3. 分页 // 是检测结果分页
        searchSourceBuilder.from((searchParam.getPageNum()-1)*EsConstant.PRODUCT_PAGESIZE);
        searchSourceBuilder.size(EsConstant.PRODUCT_PAGESIZE);//4. 高亮highlightif(!StringUtils.isEmpty(searchParam.getKeyword())){HighlightBuilder highlightBuilder =newHighlightBuilder();
            highlightBuilder.field("skuTitle");
            highlightBuilder.preTags("<b style='color:red'>");
            highlightBuilder.postTags("</b>");
            searchSourceBuilder.highlighter(highlightBuilder);}//5. 聚合//5.1 按照brand聚合TermsAggregationBuilder brandAgg =AggregationBuilders.terms("brandAgg").field("brandId");TermsAggregationBuilder brandNameAgg =AggregationBuilders.terms("brandNameAgg").field("brandName");TermsAggregationBuilder brandImgAgg =AggregationBuilders.terms("brandImgAgg").field("brandImg");//通过子聚合的方式就可以获取brand的中文名和图片了!!!
        brandAgg.subAggregation(brandNameAgg);
        brandAgg.subAggregation(brandImgAgg);
        searchSourceBuilder.aggregation(brandAgg);//5.2 按照catalog聚合TermsAggregationBuilder catalogAgg =AggregationBuilders.terms("catalogAgg").field("catalogId");// 子聚合TermsAggregationBuilder catalogNameAgg =AggregationBuilders.terms("catalogNameAgg").field("catalogName");
        catalogAgg.subAggregation(catalogNameAgg);
        searchSourceBuilder.aggregation(catalogAgg);//5.3 按照attrs聚合  嵌入式属性使用嵌入式聚合语法NestedAggregationBuilder nestedAggregationBuilder =newNestedAggregationBuilder("attrs","attrs");//按照attrId聚合     //按照attrId聚合之后再按照attrName和attrValue聚合TermsAggregationBuilder attrIdAgg    =AggregationBuilders.terms("attrIdAgg").field("attrs.attrId");TermsAggregationBuilder attrNameAgg  =AggregationBuilders.terms("attrNameAgg").field("attrs.attrName");TermsAggregationBuilder attrValueAgg =AggregationBuilders.terms("attrValueAgg").field("attrs.attrValue");
        attrIdAgg.subAggregation(attrNameAgg);
        attrIdAgg.subAggregation(attrValueAgg);

        nestedAggregationBuilder.subAggregation(attrIdAgg);
        searchSourceBuilder.aggregation(nestedAggregationBuilder);

        log.debug("构建的DSL语句 {}",searchSourceBuilder.toString());SearchRequest request =newSearchRequest(newString[]{EsConstant.PRODUCT_INDEX}, searchSourceBuilder);return request;}/**
     * 将ES响应数据封装成结果
     * @param searchParam
     * @param searchResponse
     * @return
     */privateSearchResultbulidSearchResult(SearchParam searchParam,SearchResponse searchResponse){SearchResult result =newSearchResult();SearchHits hits = searchResponse.getHits();//1. 封装查询到的商品信息if(hits.getHits()!=null&&hits.getHits().length>0){List<SkuEsModel> skuEsModels =newArrayList<>();for(SearchHit hit : hits){String sourceAsString = hit.getSourceAsString();SkuEsModel skuEsModel = JSON.parseObject(sourceAsString,SkuEsModel.class);//设置高亮属性if(!StringUtils.isEmpty(searchParam.getKeyword())){HighlightField skuTitle = hit.getHighlightFields().get("skuTitle");String highLight = skuTitle.getFragments()[0].string();
                    skuEsModel.setSkuTitle(highLight);}
                skuEsModels.add(skuEsModel);}
            result.setProducts(skuEsModels);}//2. 封装分页信息//2.1 当前页码
        result.setPageNum(searchParam.getPageNum());//2.2 总记录数long total = hits.getTotalHits().value;
        result.setTotal(total);//2.3 总页码Integer totalPages =(int)total %EsConstant.PRODUCT_PAGESIZE ==0?(int)total /EsConstant.PRODUCT_PAGESIZE :(int)total /EsConstant.PRODUCT_PAGESIZE +1;
        result.setTotalPages(totalPages);List<Integer> pageNavs =newArrayList<>();for(int i =1; i <= totalPages; i++){
            pageNavs.add(i);}
        result.setPageNavs(pageNavs);//3. 查询结果涉及到的品牌List<SearchResult.BrandVo> brandVos =newArrayList<>();Aggregations aggregations = searchResponse.getAggregations();//ParsedLongTerms用于接收terms聚合的结果,并且可以把key转化为Long类型的数据ParsedLongTerms brandAgg = aggregations.get("brandAgg");for(Terms.Bucket bucket : brandAgg.getBuckets()){//3.1 得到品牌idLong brandId = bucket.getKeyAsNumber().longValue();//获取子聚合拿到brand中文名和图片Aggregations subBrandAggs = bucket.getAggregations();//3.2 得到品牌图片ParsedStringTerms brandImgAgg=subBrandAggs.get("brandImgAgg");String brandImg = brandImgAgg.getBuckets().get(0).getKeyAsString();//3.3 得到品牌名字Terms brandNameAgg=subBrandAggs.get("brandNameAgg");String brandName = brandNameAgg.getBuckets().get(0).getKeyAsString();SearchResult.BrandVo brandVo =newSearchResult.BrandVo(brandId, brandName, brandImg);
            brandVos.add(brandVo);}
        result.setBrands(brandVos);//4. 查询涉及到的所有分类List<SearchResult.CatalogVo> catalogVos =newArrayList<>();ParsedLongTerms catalogAgg = aggregations.get("catalogAgg");for(Terms.Bucket bucket : catalogAgg.getBuckets()){//4.1 获取分类idLong catalogId = bucket.getKeyAsNumber().longValue();Aggregations subcatalogAggs = bucket.getAggregations();//4.2 获取分类名ParsedStringTerms catalogNameAgg=subcatalogAggs.get("catalogNameAgg");String catalogName = catalogNameAgg.getBuckets().get(0).getKeyAsString();SearchResult.CatalogVo catalogVo =newSearchResult.CatalogVo(catalogId, catalogName);
            catalogVos.add(catalogVo);}
        result.setCatalogs(catalogVos);//5 查询涉及到的所有属性List<SearchResult.AttrVo> attrVos =newArrayList<>();//ParsedNested用于接收内置嵌入式属性的聚合ParsedNested parsedNested=aggregations.get("attrs");ParsedLongTerms attrIdAgg=parsedNested.getAggregations().get("attrIdAgg");for(Terms.Bucket bucket : attrIdAgg.getBuckets()){//5.1 查询属性idLong attrId = bucket.getKeyAsNumber().longValue();//获取子聚合Aggregations subAttrAgg = bucket.getAggregations();//5.2 查询属性名ParsedStringTerms attrNameAgg=subAttrAgg.get("attrNameAgg");String attrName = attrNameAgg.getBuckets().get(0).getKeyAsString();//5.3 查询属性值ParsedStringTerms attrValueAgg = subAttrAgg.get("attrValueAgg");List<String> attrValues =newArrayList<>();for(Terms.Bucket attrValueAggBucket : attrValueAgg.getBuckets()){String attrValue = attrValueAggBucket.getKeyAsString();
                attrValues.add(attrValue);List<SearchResult.NavVo> navVos =newArrayList<>();}SearchResult.AttrVo attrVo =newSearchResult.AttrVo(attrId, attrName, attrValues);
            attrVos.add(attrVo);}
        result.setAttrs(attrVos);return result;}}

ES集群

ELasticsearch的集群是由多个节点组成的,通过cluster.name设置集群名称,并且用于区分其它的集群,每个节点通过node.name指定节点的名称。

ES集群中的节点类型:
1、主节点
主节点负责创建索引、删除索引、分配分片、追踪集群中的节点状态等工作。ElasticSearch中的主节点的工作量相对较轻,用户的请求可以发往集群中任何一个节点,由该节点负责分发和返回结果,而不需要经过主节点转发。而主节点是由候选主节点通过ZenDiscovery机制选举出来的,所以要想成为主节点,首先要先成为候选主节点。

2、候选主节点
在ElasticSearch集群初始化或者主节点宕机的情况下,由候选主节点中选举其中一个作为主节点。指定候选主节点的配置为:node.master:true。

3、数据节点
数据节点负责数据的存储和相关具体操作,比如CRUD、搜索、聚合。所以,数据节点对机器配置要求比较高,首先需要有足够的磁盘空间来存储数据,其次数据操作对系统CPU、Memory和IO的性能消耗都很大。通常随着集群的扩大,需要增加更多的数据节点来提高可用性。指定数据节点的配置:node.data:true。
ElasticSearch是允许一个节点既做候选主节点也做数据节点的,但是数据节点的负载较重,所以需要考虑将二者分离开,设置专用的候选主节点和数据节点,避免因数据节点负责重导致主节点不响应。

4、客户端节点
客户端节点就是既不做候选主节点也不做数据节点的节点,只负责请求的分发、汇总等等,但是这样的工作,其实任何一个节点都可以完成,因为在ElasticSearch中一个集群内的节点都可以执行任何请求,其会负责将请求转发给对应的节点进行处理。所以单独增加这样的节点更多是为了负载均衡。指定该节点的配置为:
node.master:false
node.data:false

分片
为了将数据添加到Elasticsearch,我们需要索引(index)——一个存储关联数据的地方。实际上,索引只是一个用来指向一个或多个分片(shards)的“逻辑命名空间(logical namespace)”.
在这里插入图片描述

集群新增节点

  • 向集群增加一个节点前后,索引发生了些什么。在左端,索引的主分片全部分配到节点 Node1,而副本分片没有地方分配。在这种状态下,集群是黄色的。
  • 一旦第二个节点加入,尚未分配的副本分片就会分配到新的节点 Node2,这使得集群变为了绿色的状态。
  • 当另一个节点加入的时候,Elasticsearch 会自动地尝试将分片在所有节点上进行均匀分配。在这里插入图片描述

集群参考:http://dljz.nicethemes.cn/news/show-107233.html
集群参考:https://blog.csdn.net/qq_40977118/article/details/123301013


本文转载自: https://blog.csdn.net/m0_48268301/article/details/125232549
版权归原作者 Traving Yu 所有, 如有侵权,请联系我们删除。

“高并发高可用之ElasticSearch”的评论:

还没有评论