访问日志pig作业计数和不同计数

euoag5mw  于 2021-06-25  发布在  Pig
关注(0)|答案(1)|浏览(304)

我有一些http访问日志的tbs,我正试图分析这些日志以供历史参考。我试图得到每天的独特访客(IP)和每日点击量。
如果是mysql,我会这样做:

SELECT COUNT(1) from access_logs union
SELECT COUNT(DISTINCT(ip)) from access_logs

有没有什么方法可以用一个工作做类似的事情,这样我就不用再做Map工作了?
我知道这不起作用,但这是一种功能,我正在寻找。

X = (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
Y = COUNT(X);
Z = COUNT(DISTINCT(X.IP);
OUT = UNION Y,Z;
STORE OUT into ...
9bfwbjaz

9bfwbjaz1#

如果您想计算输入中的所有记录,则需要使用group all,这将创建一个包。出于性能原因,它还使用了累加器distinct函数org.apache.pig.builtin.distinct

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
GRP = group IPs all;
OUT = foreach GRP generate COUNT(IPs) as all_cnt, COUNT(org.apache.pig.builtin.Distinct(IPs.ip)) as distinct_cnt;

如果IP太多,并且出现与内存相关的异常,则可以执行以下操作:

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
Dist_IPs = distinct IPs;
GRP_DIST = group Dist_IPs all;
DIST = foreach GRP_DIST generate COUNT(GRP_DIST) as cnt, 'dist' as category;

GRP_ALL = group IPs all;
ALL = foreach GRP_ALL generate COUNT(GRP_ALL)as cnt, 'all' as category;

OUT = union DIST, ALL;

相关问题