Go 语言高效分词, 支持英文、中文、日文等

该版本主要增加: AlphaNum 拉丁字母和数字分词设置, RemoveToken, gse.New(), 分词 ToLower 设置; Tirm 清除标点符号和空格, TextFreq 设置默认加载 frenquency, cutDAGNoHMM 和最短路径加动态规划配置, 支持 ", " 分割字典文件等功能.

增加更多示例、测试和 benchmark 代码, 修复 bug.

项目地址: https://github.com/go-ego/gse

Nisqually Glacier

Add

add alphanum split words support use variable to control default frequency

add gse.New() function and update test code, add New() function AlphaNum set support

add ToLower set support and update code

add remove token support and test code, add addToken and removeToken benchmark test, update benchmark, example.go and godoc add more test code

add English example code, Add English token test code and Update README.md

update segment test code update segment test code, add more test and remove unless code

add Trim punct and space function support, add Trim test code, add Trim benchmark test code, add Trim example code

update Trim function not exclude "include other char string"

update DictPaths() allow use ", " to separated dictionary files

add cutDAGNoHMM support and test code, add Cut() cutDAGNoHMM option support, add test and benchmark code

add python and node binding

Updtae

update gse code export splitTextToWords function export lookupTokens function

update example code update data re base code

add Find token example code update English example code

update example code use mixed text update dictionary example code

update code fixed example print update hmm test and example code

update gse server code update server code add hmm support, update jquery.min.js to v3.5.0 update server.go listen log print

update code and README.md Update README.md, add binding...

update godoc update load model godoc update godoc and var name

update go mod pkg and version remove unless go.sum hash update go mod and remove vendor

add github workflow support update go.yml update and fmt circle.yml, travis.yml add CI go1.13x support remove go1.11.x

rename utils.go to seg_utils.go

add more benchmark support add hmm benchmark test support add segmenter benchmark test code

add idf and pos files update issue and pull request template

update segment test code use expect update alpha num test code