Skip to content

PyTorch library for synthesizing programs from natural language

License

Notifications You must be signed in to change notification settings

HiroakiMikami/mlprogram

Repository files navigation

mlprogram

A Library of Deep Learning (Machine Learning) for Programming Tasks. It provides a toolbox for implementing and evaluating deep learning methods related to programming.

Purpose

The main purpose of this repository is making my experiments easy. Recently, many papers proposed deep learning methods for programming, such as programming by example and auto reparing. But many of them requires complex and error-prone implementations. For example, beam search decoding with a programming language grammar is complex and unique to this task. I want to create and maintain well-tested implementations of such algorithms.

Focuses

  • Library for handling programming languages in deep learning tasks
  • Utilities for benchmark datasets of various tasks
  • Simple baseline solution for program generation

Now I do not place value on re-implementing exsiting papers. The machine learning for programming field is still immature. There are no de-fact benchmark tasks in this field (such as image classification w/ ImageNet and object detection w/ COCO in the image). Also, there are no de-fact model (such as ResNet in the image).

Feature Lists and Plans

Benchmark

NL2Prog (Hearthstone)

Method #params [MiB] training time [min] max time per example [sec] BLEU@top1 config name
tree LSTM 7.7 92 15 0.75020 hearthstone/baseline_evaluate_short
tree LSTM 7.7 92 180 0.76540 hearthstone/baseline_evaluate_long

Programming by Example without Inputs (CSG)

Method #params [MiB] training time [min] max time per example [sec] generation rate config file
tree LSTM 16 75 30 18/30 csg/baseline_evaluate_short
tree LSTM 16 75 360 22/30 csg/baseline_evaluate_long
tree LSTM + REINFORCESynthesizer 16 75 30 18/30 csg/baseline_evaluate_rl_synthesizer_short
tree LSTM + REINFORCESynthesizer 16 75 360 22/30 csg/baseline_evaluate_rl_synthesizer_short

Auto Repair

TODO

Usage Examples

tools/launch.py is the launcher script and configs directory contains the examples.

Train/Evaluate NL2Code with Hearthstone Dataset

It requires CUDA enabled GPU.

$ python tools/launch.py --config configs/nl2code/nl2code_train.py
$ python tools/launch.py --config configs/nl2code/nl2code_evaluate.py

Warning

  • The implementation is highly experimental, and I may change it significantly.
  • The reproduced algorithms may be different from the authors' implementations. For example, the original implementation of NL2Code uses the grammar of Python 2.7.x while this repository uses the grammar of running Python version.