wjjleopard 写了：写了一个shell脚本，主要是sed和awk组合，处理一个1W多行的文件
竟然用了5分钟。
是本来shell性能就这样还是可能我写法有问题？
之前一直用java处理文件，想尝试下shell，却遇到这个问题。在这里请教大家下
[bash]#!/bin/bash
#******************************************************
#
# 本脚本的目的是为了从简版EPG的access log中提取
# 时间，ChannelID，loginName和ip
# 生成一个CSV文件，供方便入库
#
# version 0.1 2011-09-27
# author wjj
#******************************************************
format(){
if [ ! -d "$1" ]; then
echo ""
echo ""
echo "process "$1
echo "beginTime:"`date`
logFilePath=${newFolderName}/`basename $1`
#rename the file
logFilePath=${logFilePath%.*}.csv
echo $logFilePath
#check new log file exist
if [ -e "$logFilePath" ]; then
> $logFilePath
else
touch $logFilePath
fi
tempFileName="temp.temp.temp"
#process each line
rowCount=`grep "/simpleEPG/play.jsp" $1 | wc -l`
echo "row counts:"${rowCount} " " $1
#write lines to new file
grep "/simpleEPG/play.jsp" $1 > $tempFileName
count=1
#while [ $count -le $rowCount ]
for row in `cat $tempFileName`
do
#row=`sed -n ${count}p $tempFileName`
myDate=`echo $row | egrep -o '\[.*\]' | awk '{print $1}' | awk 'BEGIN {FS="["} {print $2}'`
channelID=`echo $row | egrep -o '".*"'| awk '{print $2}' | awk 'BEGIN {FS="?"} {print $2}' | awk 'BEGIN {FS="&"} {print $1}' | awk 'BEGIN {FS="="} {print $2}'`
loginName=`echo $row | egrep -o '".*"'| awk '{print $2}' | awk 'BEGIN {FS="?"} {print $2}' | awk 'BEGIN {FS="&"} {print $2}' | awk 'BEGIN {FS="="} {print $2}'`
ip=`echo $row | egrep -o '".*"'| awk '{print $2}' | awk 'BEGIN {FS="?"} {print $2}' | awk 'BEGIN {FS="&"} {print $3}' | awk 'BEGIN {FS="="} {print $2}'`
#write new line to file
echo ${myDate},${channelID},${loginName},${ip} >> $logFilePath
count=$(($count+1))
done
#movei processed file to old folder
mv $1 $oldFolderName
# echo message
echo "endTime: "`date`
echo $1 "processed!"
else
echo "folder $1 ignored!"
fi
}
#exit on any error
set -e
# for simple,the only parameter is the log folder's path
logPath=$1
# check paramter
if [ -z "$logPath" ]; then
echo "usage: $0 logFolderPath"
exit 1
fi
# check folder existing
if [ ! -d "$logPath" ]; then
echo "$logPath not a valid folder"
exit 1
fi
# format the logPath
logPath=`dirname $1`/`basename $1`
# if necessary,mkdir new folder and backup folder
newFolderName=$logPath"/"`date +%F`"-new"
oldFolderName=$logPath"/"`date +%F`"-old"
#echo $newFolderName $oldFolderName
if [ ! -d "$newFolderName" ]; then
mkdir $newFolderName
fi
if [ ! -d "$oldFolderName" ]; then
mkdir $oldFolderName
fi
# format the log files
for logFile in ${logPath}/*; do
format $logFile;
done
[/bash]
我给你贴出来，grep次数很多不必要的，awk可以匹配正则，sed也可以，awk我也不知道用管道有什么缺点，不过还是写成脚本好点。