《電子技術(shù)應(yīng)用》
您所在的位置:首頁 > 人工智能 > 設(shè)計(jì)應(yīng)用 > 一種基于狀態(tài)預(yù)測的多線程數(shù)據(jù)過濾算法
一種基于狀態(tài)預(yù)測的多線程數(shù)據(jù)過濾算法
電子技術(shù)應(yīng)用
楊嘉佳,李正,鄭兒,姚旺君,趙靜,關(guān)健
中國電子信息產(chǎn)業(yè)集團(tuán)有限公司第六研究所
摘要: 數(shù)據(jù)過濾算法在大數(shù)據(jù)處理領(lǐng)域有著重要的作用。基于正則表達(dá)式匹配技術(shù)的數(shù)據(jù)過濾算法憑借強(qiáng)大的特征表達(dá)能力適合于處理大規(guī)模復(fù)雜數(shù)據(jù)。然而,傳統(tǒng)的正則表達(dá)式匹配過程為串行匹配,造成性能低,無法滿足現(xiàn)代數(shù)據(jù)處理的需求。針對傳統(tǒng)正則表達(dá)式匹配性能低的問題,提出一種基于多線程和狀態(tài)預(yù)測的正則表達(dá)式加速匹配算法,稱之為μFA:基于向量指令執(zhí)行字符值比較,獲取可直接跳過的信任字符數(shù)。同時,基于多線程加速和狀態(tài)猜測技術(shù),實(shí)現(xiàn)字符串的分段匹配處理,通過圈定字符危險區(qū)域,研判各分段最終匹配結(jié)果的正確性。實(shí)驗(yàn)結(jié)果表明,μFA算法的吞吐率是原始DFA算法的10.12~91.36倍、ßFA算法的1.08~2.97倍。
中圖分類號:TP391.1 文獻(xiàn)標(biāo)志碼:A DOI: 10.16157/j.issn.0258-7998.245321
中文引用格式: 楊嘉佳,李正,鄭兒,等. 一種基于狀態(tài)預(yù)測的多線程數(shù)據(jù)過濾算法[J]. 電子技術(shù)應(yīng)用,2024,50(12):87-91.
英文引用格式: Yang Jiajia,Li Zheng,Zheng Er,et al. An accelerated regular expression matching algorithm based on multi-threading and state prediction[J]. Application of Electronic Technique,2024,50(12):87-91.
An accelerated regular expression matching algorithm based on multi-threading and state prediction
Yang Jiajia,Li Zheng,Zheng Er,Yao Wangjun,Zhao Jing,Guan Jian
The Sixth Research Institute of China Electronics Corporation
Abstract: Data filtering algorithms play a crucial role in the field of big data processing. Data filtering algorithms based on regular expression matching technology are suitable for processing large-scale complex data due to their powerful feature expression capabilities. However, the traditional regular expression matching process is serial matching, resulting in low performance that cannot meet the needs of modern data processing. To address the issue of low performance in traditional regular expression matching, an accelerated regular expression matching algorithm based on multithreading and state prediction is proposed, named μFA. This algorithm performs character value comparison based on vector instructions to obtain the number of trusted characters that can be skipped directly. Simultaneously, it utilizes multithreading acceleration and state prediction techniques to achieve segmented matching processing of strings. By delimiting dangerous character regions, it determines the correctness of the final matching results for each segment. Experimental results show that the throughput is 10.12 to 91.36 times higher than the original DFA algorithm and 1.08 to 2.97 times higher than the ßFA algorithm.
Key words : regular expression matching;state prediction;data filtering

引言

在人工智能時代[1],正則表達(dá)式匹配技術(shù)有助于數(shù)據(jù)的預(yù)處理過濾,可為業(yè)務(wù)應(yīng)用提供更高質(zhì)量的數(shù)據(jù)。例如,正則表達(dá)式規(guī)則由于其展現(xiàn)出強(qiáng)大的表征能力,可從大規(guī)模數(shù)據(jù)中過濾出復(fù)雜且符合深度學(xué)習(xí)模型要求的數(shù)據(jù),提升模型的推理精度。

數(shù)據(jù)預(yù)處理吞吐率是衡量過濾算法的重要性能因素之一,反映出在特定環(huán)境下算法可以運(yùn)行的性能極限,決定其是否適用于高性能大數(shù)據(jù)預(yù)處理領(lǐng)域。因此,本文重點(diǎn)研究如何提高基于正則表達(dá)式匹配的數(shù)據(jù)過濾性能。

當(dāng)前,已涌現(xiàn)出許多優(yōu)秀的基于正則表達(dá)式技術(shù)的數(shù)據(jù)過濾算法[2],包括基于非確定型有限自動機(jī)(Nondeterministic Finite Automata, NFA)、基于確定型有限自動機(jī)(Deterministic Finite Automata, DFA)和基于混合自動機(jī)(Hybrid Finite Automata, Hybrid-FA)等實(shí)現(xiàn)方式。其中,因DFA的數(shù)據(jù)過濾性能較為穩(wěn)定,備受研究人員和開發(fā)人員的青睞。

然而,現(xiàn)有的正則表達(dá)式過濾算法性能較低,無法滿足大數(shù)據(jù)背景下的高性能過濾需求。因此,本文提出一種基于狀態(tài)預(yù)測的多線程數(shù)據(jù)過濾算法:通過向量指令字符值比較、多線程加速、狀態(tài)猜測等技術(shù),實(shí)現(xiàn)字符串的分段匹配處理,從而提高算法的吞吐率。


本文詳細(xì)內(nèi)容請下載:

http://theprogrammingfactory.com/resource/share/2000006254


作者信息:

楊嘉佳,李正,鄭兒,姚旺君,趙靜,關(guān)健

(中國電子信息產(chǎn)業(yè)集團(tuán)有限公司第六研究所,北京 100083)


Magazine.Subscription.jpg

此內(nèi)容為AET網(wǎng)站原創(chuàng),未經(jīng)授權(quán)禁止轉(zhuǎn)載。