python中logging模塊詳解，python log文件處理_Python 處理大Log文件

最近研究 process monitor，對于取得的Log 需要進行簡單的統計得出操作的類型，結果是一個 620000行左右的文件。數據示例如下:

"15:20:33.9935624","cmd.exe","1784","IRP_MJ_CREATE","C:\EDK","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"

python中logging模塊詳解，"15:20:33.9935792","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK\build.*","SUCCESS","Type: QueryDirectory, Filter: build.*, 2: Build"

"15:20:33.9936119","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK","NO MORE FILES","Type: QueryDirectory"

"15:20:33.9936272","cmd.exe","1784","IRP_MJ_CLEANUP","C:\EDK","SUCCESS",""

range函數python？"15:20:33.9936306","cmd.exe","1784","IRP_MJ_CLOSE","C:\EDK","SUCCESS",""

"15:20:33.9937049","cmd.exe","1784","IRP_MJ_CREATE","C:\EDK","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"

"15:20:33.9937169","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK\build","SUCCESS","Type: QueryDirectory, Filter: build, 2: Build"

python csv文件。"15:20:33.9937342","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK","NO MORE FILES","Type: QueryDirectory"

"15:20:33.9937476","cmd.exe","1784","IRP_MJ_CLEANUP","C:\EDK","SUCCESS",""

"15:20:33.9937541","cmd.exe","1784","IRP_MJ_CLOSE","C:\EDK","SUCCESS",""

python3？"15:20:33.9938281","cmd.exe","1784","IRP_MJ_CREATE","C:\EDK\BaseTools\Bin","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"

"15:20:33.9938586","cmd.exe","1784","IRP_MJ_DIRECTORY_CONTROL","C:\EDK\BaseTools\Bin\build.*","NO SUCH FILE","Type: QueryDirectory, Filter: build.*"

"15:20:33.9938787","cmd.exe","1784","IRP_MJ_CLEANUP","C:\EDK\BaseTools\Bin","SUCCESS",""

python列表。我需要提取出來IRP_MJ_DIRECTORY_CONTROL 和IRP_MJ_CLEANUP這樣的操作記錄下來。

聽說Python擅長于此，于是簡單學習一下進行統計。

第一個問題是如何處理大文件(目前的數據不算大，200MB),在【參考1】找到方法；

Python logging，第二個問題是如何進行統計的問題。我只需要記錄一個操作有還是沒有，所以使用字典類型最合適不過；

最終代碼如下(Python2.7環境下運行)

class Load_Corpus_with_Iteration(object):

def __init__(self,path):

self.path=path

def __iter__(self):

for line in open(self.path):

yield line.split()

corpus = Load_Corpus_with_Iteration('logfile.csv')

operate = {}

index=0

for item in corpus:

list1 = (item[0].split(','))

opStr=list1[3]

operate[opStr]=1

index=index+1

if index % 10000 ==0:

print index,str(operate)

結果如下(為了便于閱讀，經過簡單排版)

620000 {

'"IRP_MJ_DIRECTORY_CONTROL"': 1,

'"FASTIO_READ"': 1,

'"IRP_MJ_READ"': 1,

'"FASTIO_LOCK"': 1,

'"FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION"': 1,

'"IRP_MJ_CLOSE"': 1,

'"IRP_MJ_QUERY_INFORMATION"': 1,

'"IRP_MJ_SET_INFORMATION"': 1,

'"FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION"': 1,

'"FASTIO_QUERY_INFORMATION"': 1,

'"IRP_MJ_WRITE"': 1,

'"FASTIO_ACQUIRE_FOR_CC_FLUSH"': 1,

'"IRP_MJ_FILE_SYSTEM_CONTROL"': 1,

'"IRP_MJ_QUERY_VOLUME_INFORMATION"': 1,

'"FASTIO_WRITE"': 1,

'"IRP_MJ_CREATE"': 1,

'"FASTIO_NETWORK_QUERY_OPEN"': 1,

'"FASTIO_RELEASE_FOR_CC_FLUSH"': 1,

'"FASTIO_UNLOCK_SINGLE"': 1, '

"IRP_MJ_CLEANUP"': 1,

'"FASTIO_CHECK_IF_POSSIBLE"': 1}

參考：

1. https://blog.csdn.net/chixujohnny/article/details/53069988