0
0

大数据-131 - Flink CEP 案例：检测交易活跃用户、超时未交付

点一下关注吧！！！非常感谢！！持续更新！！！

目前已经更新到了：

  • Hadoop（已更完）
  • HDFS（已更完）
  • MapReduce（已更完）
  • Hive（已更完）
  • Flume（已更完）
  • Sqoop（已更完）
  • Zookeeper（已更完）
  • HBase（已更完）
  • Redis （已更完）
  • Kafka（已更完）
  • Spark（已更完）
  • Flink（正在更新！）

章节内容

上节我们完成了如下的内容：

  • Flink CEP 开发的流程
  • CEP 开发依赖
  • CEP 案例：恶意登录检测实现

在这里插入图片描述

Fline CEP

之前已经介绍过，但是防止大家没看到，这里再简单介绍以下。

基本概念

Flink CEP（Complex Event Processing）是Apache Flink提供的一个扩展库，用于实时复杂事件处理。通过Flink CEP，开发者可以从流数据中识别出特定的事件模式。这在欺诈检测、网络安全、实时监控、物联网等场景中非常有用。

Flink CEP的核心是通过定义事件模式，从流中检测复杂事件序列。
具体来说，CEP允许用户：

  • 定义事件模式：用户可以描述感兴趣的事件组合（如连续事件、延迟事件等）。
  • 匹配模式：Flink CEP从流中搜索与定义模式相匹配的事件序列。
  • 处理匹配结果：一旦找到符合模式的事件序列，用户可以定义如何处理这些匹配。

基本组成部分

  • Pattern（模式）：描述要在事件流中匹配的事件序列。可以是单个事件或多个事件的组合。常用的模式操作包括next（紧邻）、followedBy（接续）等。
  • PatternStream（模式流）：通过应用模式定义，将事件流转变为模式流。
  • Select函数：用于从模式流中提取匹配的事件序列

CEP开发步骤

开发Flink CEP应用的基本步骤包括：

定义事件流：创建一个DataStream，表示原始的事件流。
定义事件模式：使用Flink CEP的API定义事件模式，例如连续事件、迟到事件等。
将模式应用到流中：将定义好的模式应用到事件流上，生成模式流PatternStream。
提取匹配事件：使用select函数提取匹配模式的事件，并定义如何处理这些事件。

使用场景

  • 欺诈检测：可以通过CEP识别连续发生的异常行为，如频繁的登录尝试等。
  • 网络监控：检测一段时间内的特定网络攻击模式。
  • 物联网：分析传感器数据，检测设备异常、温度异常等。
  • 用户行为分析：分析用户在某一时间段内的行为序列，从而作出预测或检测异常。

案例2：检测交易活跃用户

业务需求

业务上需要找出24小时内，至少5次有效交易的用户。
数据源如下：

new CepActiveUserBean("100XX", 0.0D, 1597905234000L),
new CepActiveUserBean("100XX", 100.0D, 1597905235000L),
new CepActiveUserBean("100XX", 200.0D, 1597905236000L),
new CepActiveUserBean("100XX", 300.0D, 1597905237000L),
new CepActiveUserBean("100XX", 400.0D, 1597905238000L),
new CepActiveUserBean("100XX", 500.0D, 1597905239000L),
new CepActiveUserBean("101XX", 0.0D, 1597905240000L),
new CepActiveUserBean("101XX", 100.0D, 1597905241000L)
  • 获取数据源
  • Watermark转化
  • keyBy转化
  • 至少5次：timeOrMore(5)
  • 24小时之内：within(Time.hours(24))
  • 模式匹配
  • 提取匹配成功的数据

编写代码

packageicu.wzk;importorg.apache.flink.api.common.eventtime.*;importorg.apache.flink.api.java.functions.KeySelector;importorg.apache.flink.cep.CEP;importorg.apache.flink.cep.PatternStream;importorg.apache.flink.cep.functions.PatternProcessFunction;importorg.apache.flink.cep.pattern.Pattern;importorg.apache.flink.cep.pattern.conditions.SimpleCondition;importorg.apache.flink.streaming.api.TimeCharacteristic;importorg.apache.flink.streaming.api.datastream.DataStreamSource;importorg.apache.flink.streaming.api.datastream.KeyedStream;importorg.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;importorg.apache.flink.streaming.api.environment.StreamExecutionEnvironment;importorg.apache.flink.streaming.api.windowing.time.Time;importorg.apache.flink.util.Collector;importjava.util.List;importjava.util.Map;publicclassFlinkCepActiveUser{publicstaticvoidmain(String[] args)throwsException{StreamExecutionEnvironment env =StreamExecutionEnvironment.getExecutionEnvironment();
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        env.setParallelism(1);DataStreamSource<CepActiveUserBean> data = env.fromElements(newCepActiveUserBean("100XX",0.0D,1597905234000L),newCepActiveUserBean("100XX",100.0D,1597905235000L),newCepActiveUserBean("100XX",200.0D,1597905236000L),newCepActiveUserBean("100XX",300.0D,1597905237000L),newCepActiveUserBean("100XX",400.0D,1597905238000L),newCepActiveUserBean("100XX",500.0D,1597905239000L),newCepActiveUserBean("101XX",0.0D,1597905240000L),newCepActiveUserBean("101XX",100.0D,1597905241000L));SingleOutputStreamOperator<CepActiveUserBean> watermark = data
                .assignTimestampsAndWatermarks(newWatermarkStrategy<CepActiveUserBean>(){@OverridepublicWatermarkGenerator<CepActiveUserBean>createWatermarkGenerator(WatermarkGeneratorSupplier.Context context){returnnewWatermarkGenerator<CepActiveUserBean>(){long maxTimestamp =Long.MAX_VALUE;long maxOutOfOrderness =500L;@OverridepublicvoidonEvent(CepActiveUserBean event,long eventTimestamp,WatermarkOutput output){
                                maxTimestamp =Math.max(event.getTimestamp(), maxTimestamp);}@OverridepublicvoidonPeriodicEmit(WatermarkOutput output){
                                output.emitWatermark(newWatermark(maxTimestamp - maxOutOfOrderness));}};}}.withTimestampAssigner((element, recordTimes)-> element.getTimestamp()));KeyedStream<CepActiveUserBean,String> keyed = watermark
                .keyBy(newKeySelector<CepActiveUserBean,String>(){@OverridepublicStringgetKey(CepActiveUserBean value)throwsException{return value.getUsername();}});Pattern<CepActiveUserBean,CepActiveUserBean> pattern =Pattern.<CepActiveUserBean>begin("start").where(newSimpleCondition<CepActiveUserBean>(){@Overridepublicbooleanfilter(CepActiveUserBean value)throwsException{return value.getPrice()>0;}}).timesOrMore(5).within(Time.hours(24));PatternStream<CepActiveUserBean> parentStream =CEP.pattern(keyed, pattern);SingleOutputStreamOperator<CepActiveUserBean> process = parentStream
                .process(newPatternProcessFunction<CepActiveUserBean,CepActiveUserBean>(){@OverridepublicvoidprocessMatch(Map<String,List<CepActiveUserBean>> map,Context context,Collector<CepActiveUserBean> collector)throwsException{System.out.println("map: "+ map);}});
        process.print();
        env.execute("FlinkCepActiveUser");}}classCepActiveUserBean{privateString username;privateDouble price;privateLong timestamp;publicCepActiveUserBean(String username,Double price,Long timestamp){this.username = username;this.price = price;this.timestamp = timestamp;}publicStringgetUsername(){return username;}publicvoidsetUsername(String username){this.username = username;}publicDoublegetPrice(){return price;}publicvoidsetPrice(Double price){this.price = price;}publicLonggetTimestamp(){return timestamp;}publicvoidsetTimestamp(Long timestamp){this.timestamp = timestamp;}@OverridepublicStringtoString(){return"CepActiveUserBean{"+"username='"+ username +'\''+", price="+ price +", timestamp="+ timestamp +'}';}}

运行结果

map: {start=[CepActiveUserBean{username='100XX', price=100.0, timestamp=1597905235000}, CepActiveUserBean{username='100XX', price=200.0, timestamp=1597905236000}, CepActiveUserBean{username='100XX', price=300.0, timestamp=1597905237000}, CepActiveUserBean{username='100XX', price=400.0, timestamp=1597905238000}, CepActiveUserBean{username='100XX', price=500.0, timestamp=1597905239000}]}

Process finished with exit code 0

运行结果如下图所示：
在这里插入图片描述

案例3：超时未支付

业务需求

找出下单后10分钟没有支付的订单，数据源如下：

new TimeOutPayBean(1L, "create", 1597905234000L),
new TimeOutPayBean(1L, "pay", 1597905235000L),
new TimeOutPayBean(2L, "create", 1597905236000L),
new TimeOutPayBean(2L, "pay", 1597905237000L),
new TimeOutPayBean(3L, "create", 1597905239000L)
  • 获取数据源
  • 转 Watermark
  • keyBy 转化
  • 做出 Pattern （下单以后10分钟未支付）
  • 模式匹配
  • 取出匹配成功的数据

编写代码

packageicu.wzk;importorg.apache.flink.api.common.eventtime.*;importorg.apache.flink.api.java.functions.KeySelector;importorg.apache.flink.cep.CEP;importorg.apache.flink.cep.PatternSelectFunction;importorg.apache.flink.cep.PatternStream;importorg.apache.flink.cep.PatternTimeoutFunction;importorg.apache.flink.cep.pattern.Pattern;importorg.apache.flink.cep.pattern.conditions.IterativeCondition;importorg.apache.flink.streaming.api.TimeCharacteristic;importorg.apache.flink.streaming.api.datastream.DataStream;importorg.apache.flink.streaming.api.datastream.DataStreamSource;importorg.apache.flink.streaming.api.datastream.KeyedStream;importorg.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;importorg.apache.flink.streaming.api.environment.StreamExecutionEnvironment;importorg.apache.flink.streaming.api.windowing.time.Time;importorg.apache.flink.util.OutputTag;importjava.util.List;importjava.util.Map;publicclassFlinkCepTimeOutPay{publicstaticvoidmain(String[] args)throwsException{StreamExecutionEnvironment env =StreamExecutionEnvironment.getExecutionEnvironment();
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        env.setParallelism(1);DataStreamSource<TimeOutPayBean> data = env.fromElements(newTimeOutPayBean(1L,"create",1597905234000L),newTimeOutPayBean(1L,"pay",1597905235000L),newTimeOutPayBean(2L,"create",1597905236000L),newTimeOutPayBean(2L,"pay",1597905237000L),newTimeOutPayBean(3L,"create",1597905239000L));DataStream<TimeOutPayBean> watermark = data
                .assignTimestampsAndWatermarks(newWatermarkStrategy<TimeOutPayBean>(){@OverridepublicWatermarkGenerator<TimeOutPayBean>createWatermarkGenerator(WatermarkGeneratorSupplier.Context context){returnnewWatermarkGenerator<TimeOutPayBean>(){long maxTimestamp =Long.MAX_VALUE;long maxOutOfOrderness =500L;@OverridepublicvoidonEvent(TimeOutPayBean event,long eventTimestamp,WatermarkOutput output){
                                maxTimestamp =Math.max(maxTimestamp, event.getTimestamp());}@OverridepublicvoidonPeriodicEmit(WatermarkOutput output){
                                output.emitWatermark(newWatermark(maxTimestamp - maxOutOfOrderness));}};}}.withTimestampAssigner((element, recordTimestamp)-> element.getTimestamp()));KeyedStream<TimeOutPayBean,Long> keyedStream = watermark
                .keyBy(newKeySelector<TimeOutPayBean,Long>(){@OverridepublicLonggetKey(TimeOutPayBean value)throwsException{return value.getUserId();}});// 逻辑处理代码OutputTag<TimeOutPayBean> orderTimeoutOutput =newOutputTag<>("orderTimeout"){};Pattern<TimeOutPayBean,TimeOutPayBean> pattern =Pattern.<TimeOutPayBean>begin("begin").where(newIterativeCondition<TimeOutPayBean>(){@Overridepublicbooleanfilter(TimeOutPayBean timeOutPayBean,Context<TimeOutPayBean> context)throwsException{return timeOutPayBean.getOperation().equals("create");}}).followedBy("pay").where(newIterativeCondition<TimeOutPayBean>(){@Overridepublicbooleanfilter(TimeOutPayBean timeOutPayBean,Context<TimeOutPayBean> context)throwsException{return timeOutPayBean.getOperation().equals("pay");}}).within(Time.seconds(600));PatternStream<TimeOutPayBean> patternStream =CEP.pattern(keyedStream, pattern);SingleOutputStreamOperator<TimeOutPayBean> result = patternStream
                .select(orderTimeoutOutput,newPatternTimeoutFunction<TimeOutPayBean,TimeOutPayBean>(){@OverridepublicTimeOutPayBeantimeout(Map<String,List<TimeOutPayBean>> map,long l)throwsException{return map.get("begin").get(0);}},newPatternSelectFunction<TimeOutPayBean,TimeOutPayBean>(){@OverridepublicTimeOutPayBeanselect(Map<String,List<TimeOutPayBean>> map)throwsException{return map.get("pay").get(0);}});// 输出结果// result.print();System.out.println("==============");DataStream<TimeOutPayBean> sideOutput = result
                .getSideOutput(orderTimeoutOutput);
        sideOutput.print();// 执行
        env.execute("FlinkCepTimeOutPay");}}classTimeOutPayBean{privateLong userId;privateString operation;privateLong timestamp;publicTimeOutPayBean(Long userId,String operation,Long timestamp){this.userId = userId;this.operation = operation;this.timestamp = timestamp;}publicLonggetUserId(){return userId;}publicvoidsetUserId(Long userId){this.userId = userId;}publicStringgetOperation(){return operation;}publicvoidsetOperation(String operation){this.operation = operation;}publicLonggetTimestamp(){return timestamp;}publicvoidsetTimestamp(Long timestamp){this.timestamp = timestamp;}@OverridepublicStringtoString(){return"TimeOutPayBean{"+"userId="+ userId +", operation='"+ operation +'\''+", timestamp="+ timestamp +'}';}}

运行结果

控制台输出为：

==============
TimeOutPayBean{userId=1, operation='pay', timestamp=1597905235000}
TimeOutPayBean{userId=3, operation='create', timestamp=1597905239000}
TimeOutPayBean{userId=2, operation='pay', timestamp=1597905237000}

Process finished with exit code 0

对应截图如下：
在这里插入图片描述

标签： 大数据 flink java
本文转载自: https://blog.csdn.net/w776341482/article/details/142163493
版权归原作者 武子康 所有， 如有侵权，请联系我们删除。
登录后发布评论

“大数据-131 - Flink CEP 案例：检测交易活跃用户、超时未交付”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

Linux在Ubuntu系统下安装MySQL数据库（全网最详细）

chrome浏览器selenium点击下载pdf时总是提示“已阻止不安全的下载”

GuiHelper：面向Java Swing的开源GUI设计辅助工具

数据仓库面试题集&离线&实时

毕业设计项目 基于大数据的K-means广告效果分析

大数据-229 离线数仓 - ODS层的构建 Hive处理 JSON 数据处理 结构化

前端三件套(HTML + CSS + JS)

文章导航