CodeXGLUE

CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified programming language tasks covering code-code (clone detection, defect detection, cloze test, code completion, code refinement, and code-to-code translation), text-code (natural language code search, text-to-code generation), code-text (code summarization) and text-text (documentation translation) scenarios. We provide three baseline models to support these tasks, including BERT-style pre-trained model (i.e. CodeBERT) which is good at understanding problems, GPT-style pre-trained model which we call CodeGPT to support completion and generation problems, and Encoder-Decoder framework that supports sequence-to-sequence generation problems.

Overall Leaderboard

Clone Detection (Code-Code)

Defect Detection (Code-Code)

Cloze Test (Code-Code)

Code Completion (Code-Code)

Code Refinement (Code-Code)

Code Translation (Code-Code)

Type Prediction (Code-Code)

Natural Language Code Search (Text-Code)

Code Generation (Text-Code)

Code Summarization (Code-Text)

Documentation Translation (Text-Text)

CodeXGLUE Submission Instructions

Once you have built a model that meets your expectations on evaluation with the dev set, you can submit your test results to get official evaluation on the test set. To ensure the integrity of the official test results, we do not release the correct answers for test set to the public. To submit your model for official evaluation on the test set, follow the below steps:

1. Generate your prediction output for the dev set.
2. Run the official evaluation methodologies found in the task specific git repo and verify your systems are running as expected.
3. Generate your prediction output for the test set.
4. Submit the following information by emailing to codexglue@microsoft.com

Your email should include:

1. Prediction results on test set. [Required]
2. Prediction results on dev set. [Recommended]
3. Individual/Team Name: Name of the individual or the team to appear in the leaderboard. [Required]
4. Individual/Team Institution: Name of the institution of the individual or the team to appear in the leaderboard. [Optional]
5. Model code: Training code for the model. [Recommended]
6. Model information: Name of the model/technique to appear in the leaderboard. [Required]
7. Paper Information: Name, Citation, URL of the paper if model is from a published work to appear in the leaderboard. [Optional]

To avoid "P-hacking" we discourage too many submissions from the same group in a short period of time.

Relevant Links

CodeXGLUE Github CodeXGLUE Paper CodeBERT Paper CodeBERT Github

How to cite

To cite CodeXGLUE:

@article{DBLP:journals/corr/abs-2102-04664,

							  author    = {Shuai Lu and

							    Daya Guo and

							    Shuo Ren and

							    Junjie Huang and

							    Alexey Svyatkovskiy and

							    Ambrosio Blanco and

							    Colin B. Clement and

							    Dawn Drain and

							    Daxin Jiang and

							    Duyu Tang and

							    Ge Li and

							    Lidong Zhou and

							    Linjun Shou and

							    Long Zhou and

							    Michele Tufano and

							    Ming Gong and

							    Ming Zhou and

							    Nan Duan and

							    Neel Sundaresan and

							    Shao Kun Deng and

							    Shengyu Fu and

							    Shujie Liu},

							  title     = {CodeXGLUE: {A} Machine Learning Benchmark Dataset for Code Understanding

							    and Generation},

							  journal   = {CoRR},

							  volume    = {abs/2102.04664},

							  year      = {2021}

						  }

If you only use part of the CodeXGLUE dataset, please also cite each seperated dataset.