Skip to content

Neural network models for joint POS tagging and dependency parsing (CoNLL 2017-2018)

License

Notifications You must be signed in to change notification settings

datquocnguyen/jPTDP

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

29 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Neural Network Models for Joint POS Tagging and Dependency Parsing

jptdpv2

Implementations of joint models for POS tagging and dependency parsing, as described in my papers:

  1. Dat Quoc Nguyen and Karin Verspoor. 2018. An improved neural network model for joint POS tagging and dependency parsing. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 81-91. [.bib] (jPTDP v2.0)
  2. Dat Quoc Nguyen, Mark Dras and Mark Johnson. 2017. A Novel Neural Network Model for Joint POS Tagging and Graph-based Dependency Parsing. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 134-142. [.bib] (jPTDP v1.0)

This github project currently supports jPTDP v2.0, while v1.0 can be found in the release section. Please cite paper [1] when jPTDP is used to produce published results or incorporated into other software. I would highly appreciate to have your bug reports, comments and suggestions about jPTDP. As a free open-source implementation, jPTDP is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

Installation

jPTDP requires the following software packages:

  • Python 2.7

  • DyNet v2.0

    $ virtualenv -p python2.7 .DyNet
    $ source .DyNet/bin/activate
    $ pip install cython numpy
    $ pip install dynet==2.0.3
    

Once you installed the prerequisite packages above, you can clone or download (and then unzip) jPTDP. Next sections show instructions to train a new joint model for POS tagging and dependency parsing, and then to utilize a pre-trained model.

NOTE: jPTDP is also ported to run with Python 3.4+ by Santiago Castro. Also note that pre-trained models I provide in the last section would not work with this ported version (see a discussion). Thus, you may want to retrain jPTDP if using this ported version.

Train a joint model

Suppose that SOURCE_DIR is simply used to denote the source code directory. Similar to files train.conllu and dev.conllu in folder SOURCE_DIR/sample or treebanks in the Universal Dependencies (UD) project, the training and development files are formatted following 10-column data format. For training, jPTDP will only use information from columns 1 (ID), 2 (FORM), 4 (Coarse-grained POS tags---UPOSTAG), 7 (HEAD) and 8 (DEPREL).

To train a joint model for POS tagging and dependency parsing, you perform:

SOURCE_DIR$ python jPTDP.py --dynet-seed 123456789 [--dynet-mem <int>] [--epochs <int>] [--lstmdims <int>] [--lstmlayers <int>] [--hidden <int>] [--wembedding <int>] [--cembedding <int>] [--pembedding <int>] [--prevectors <path-to-pre-trained-word-embedding-file>] [--model <String>] [--params <String>] --outdir <path-to-output-directory> --train <path-to-train-file>  --dev <path-to-dev-file>

where hyper-parameters in [] are optional:

  • --dynet-mem: Specify DyNet memory in MB.
  • --epochs: Specify number of training epochs. Default value is 30.
  • --lstmdims: Specify number of BiLSTM dimensions. Default value is 128.
  • --lstmlayers: Specify number of BiLSTM layers. Default value is 2.
  • --hidden: Specify size of MLP hidden layer. Default value is 100.
  • --wembedding: Specify size of word embeddings. Default value is 100.
  • --cembedding: Specify size of character embeddings. Default value is 50.
  • --pembedding: Specify size of POS tag embeddings. Default value is 100.
  • --prevectors: Specify path to the pre-trained word embedding file for initialization. Default value is "None" (i.e. word embeddings are randomly initialized).
  • --model: Specify a name for model parameters file. Default value is "model".
  • --params: Specify a name for model hyper-parameters file. Default value is "model.params".
  • --outdir: Specify path to directory where the trained model will be saved.
  • --train: Specify path to the training data file.
  • --dev: Specify path to the development data file.

For example:

SOURCE_DIR$ python jPTDP.py --dynet-seed 123456789 --dynet-mem 1000 --epochs 30 --lstmdims 128 --lstmlayers 2 --hidden 100 --wembedding 100 --cembedding 50 --pembedding 100  --model trialmodel --params trialmodel.params --outdir sample/ --train sample/train.conllu --dev sample/dev.conllu

will produce model files trialmodel and trialmodel.params in folder SOURCE_DIR/sample.

If you would like to use the fine-grained language-specific POS tags in the 5th column instead of the coarse-grained POS tags in the 4th column, you should use swapper.py in folder SOURCE_DIR/utils to swap contents in the 4th and 5th columns:

SOURCE_DIR$ python utils/swapper.py <path-to-train-(and-dev)-file>

For example:

SOURCE_DIR$ python utils/swapper.py sample/train.conllu
SOURCE_DIR$ python utils/swapper.py sample/dev.conllu

will generate two new files for training: train.conllu.ux2xu and dev.conllu.ux2xu in folder SOURCE_DIR/sample.

Utilize a pre-trained model

Assume that you are going to utilize a pre-trained model for annotating a corpus whose each line represents a tokenized/word-segmented sentence. You should use converter.py in folder SOURCE_DIR/utils to obtain a 10-column data format of this corpus:

SOURCE_DIR$ python utils/converter.py <file-path>

For example:

SOURCE_DIR$ python utils/converter.py sample/test

will generate in folder SOURCE_DIR/sample a file named test.conllu which can be used later as input to the pre-trained model.

To utilize a pre-trained model for POS tagging and dependency parsing, you perform:

SOURCE_DIR$ python jPTDP.py --predict --model <path-to-model-parameters-file> --params <path-to-model-hyper-parameters-file> --test <path-to-10-column-input-file> --outdir <path-to-output-directory> --output <String>
  • --model: Specify path to model parameters file.
  • --params: Specify path to model hyper-parameters file.
  • --test: Specify path to 10-column input file.
  • --outdir: Specify path to directory where output file will be saved.
  • --output: Specify name of the output file.

For example:

SOURCE_DIR$ python jPTDP.py --predict --model sample/trialmodel --params sample/trialmodel.params --test sample/test.conllu --outdir sample/ --output test.conllu.pred
SOURCE_DIR$ python jPTDP.py --predict --model sample/trialmodel --params sample/trialmodel.params --test sample/dev.conllu --outdir sample/ --output dev.conllu.pred

will produce output files test.conllu.pred and dev.conllu.pred in folder SOURCE_DIR/sample.

Pre-trained models

Pre-trained jPTDP v2.0 models, which were trained on English WSJ Penn treebank, GENIA and UD v2.2 treebanks, can be found at HERE. Results on test sets (as detailed in paper [1]) are as follows:

Treebank Model name POS UAS LAS
English WSJ Penn treebank model256 97.97 94.51 92.87
English WSJ Penn treebank model 97.88 94.25 92.58

model256 and model denote the pre-trained models which use 256- and 128-dimensional LSTM hidden states, respectively, i.e. model256 is more accurate but slower.

Treebank Code UPOS UAS LAS
UD_Afrikaans-AfriBooms af_afribooms 95.73 82.57 78.89
UD_Ancient_Greek-PROIEL grc_proiel 96.05 77.57 72.84
UD_Ancient_Greek-Perseus grc_perseus 88.95 65.09 58.35
UD_Arabic-PADT ar_padt 96.33 86.08 80.97
UD_Basque-BDT eu_bdt 93.62 79.86 75.07
UD_Bulgarian-BTB bg_btb 98.07 91.47 87.69
UD_Catalan-AnCora ca_ancora 98.46 90.78 88.40
UD_Chinese-GSD zh_gsd 93.26 82.50 77.51
UD_Croatian-SET hr_set 97.42 88.74 83.62
UD_Czech-CAC cs_cac 98.87 89.85 87.13
UD_Czech-FicTree cs_fictree 97.98 88.94 85.64
UD_Czech-PDT cs_pdt 98.74 89.64 87.04
UD_Czech-PUD cs_pud 96.71 87.62 82.28
UD_Danish-DDT da_ddt 96.18 82.17 78.88
UD_Dutch-Alpino nl_alpino 95.62 86.34 82.37
UD_Dutch-LassySmall nl_lassysmall 95.21 86.46 82.14
UD_English-EWT en_ewt 95.48 87.55 84.71
UD_English-GUM en_gum 94.10 84.88 80.45
UD_English-LinES en_lines 95.55 80.34 75.40
UD_English-PUD en_pud 95.25 87.49 84.25
UD_Estonian-EDT et_edt 96.87 85.45 82.13
UD_Finnish-FTB fi_ftb 94.53 86.10 82.45
UD_Finnish-PUD fi_pud 96.44 87.54 84.60
UD_Finnish-TDT fi_tdt 96.12 86.07 82.92
UD_French-GSD fr_gsd 97.11 89.45 86.43
UD_French-Sequoia fr_sequoia 97.92 89.71 87.43
UD_French-Spoken fr_spoken 94.25 79.80 73.45
UD_Galician-CTG gl_ctg 97.12 85.09 81.93
UD_Galician-TreeGal gl_treegal 93.66 77.71 71.63
UD_German-GSD de_gsd 94.07 81.45 76.68
UD_Gothic-PROIEL got_proiel 93.45 79.80 71.85
UD_Greek-GDT el_gdt 96.59 87.52 84.64
UD_Hebrew-HTB he_htb 96.24 87.65 82.64
UD_Hindi-HDTB hi_hdtb 96.94 93.25 89.83
UD_Hungarian-Szeged hu_szeged 92.07 76.18 69.75
UD_Indonesian-GSD id_gsd 93.29 84.64 77.71
UD_Irish-IDT ga_idt 89.74 75.72 65.78
UD_Italian-ISDT it_isdt 98.01 92.33 90.20
UD_Italian-PoSTWITA it_postwita 95.41 84.20 79.11
UD_Japanese-GSD ja_gsd 97.27 94.21 92.02
UD_Japanese-Modern ja_modern 70.53 66.88 49.51
UD_Korean-GSD ko_gsd 93.35 81.32 76.58
UD_Korean-Kaist ko_kaist 93.53 83.59 80.74
UD_Latin-ITTB la_ittb 98.12 82.99 79.96
UD_Latin-PROIEL la_proiel 95.54 74.95 69.76
UD_Latin-Perseus la_perseus 82.36 57.21 46.28
UD_Latvian-LVTB lv_lvtb 93.53 81.06 76.13
UD_North_Sami-Giella sme_giella 87.48 65.79 58.09
UD_Norwegian-Bokmaal no_bokmaal 97.73 89.83 87.57
UD_Norwegian-Nynorsk no_nynorsk 97.33 89.73 87.29
UD_Norwegian-NynorskLIA no_nynorsklia 85.22 64.14 54.31
UD_Old_Church_Slavonic-PROIEL cu_proiel 93.69 80.59 73.93
UD_Old_French-SRCMF fro_srcmf 95.12 86.65 81.15
UD_Persian-Seraji fa_seraji 96.66 88.07 84.07
UD_Polish-LFG pl_lfg 98.22 95.29 93.10
UD_Polish-SZ pl_sz 97.05 90.98 87.66
UD_Portuguese-Bosque pt_bosque 96.76 88.67 85.71
UD_Romanian-RRT ro_rrt 97.43 88.74 83.54
UD_Russian-SynTagRus ru_syntagrus 98.51 91.00 88.91
UD_Russian-Taiga ru_taiga 85.49 65.52 56.33
UD_Serbian-SET sr_set 97.40 89.32 85.03
UD_Slovak-SNK sk_snk 95.18 85.88 81.89
UD_Slovenian-SSJ sl_ssj 97.79 88.26 86.10
UD_Slovenian-SST sl_sst 89.50 66.14 58.13
UD_Spanish-AnCora es_ancora 98.57 90.30 87.98
UD_Swedish-LinES sv_lines 95.51 83.60 78.97
UD_Swedish-PUD sv_pud 92.10 79.53 74.53
UD_Swedish-Talbanken sv_talbanken 96.55 86.53 83.01
UD_Turkish-IMST tr_imst 92.93 70.53 62.55
UD_Ukrainian-IU uk_iu 95.24 83.47 79.38
UD_Urdu-UDTB ur_udtb 93.35 86.74 80.44
UD_Uyghur-UDT ug_udt 87.63 76.14 63.37
UD_Vietnamese-VTB vi_vtb 87.63 67.72 58.27