File size: 1,554 Bytes
3eab653 583474c 3eab653 0da9ca2 0dd993b 3eab653 0dd993b 3eab653 0dd993b 0b3dd7b 5d717a5 65bcc25 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
---
tags:
- flan
- opt
- peft
datasets:
- SirNeural/flan_v2
metrics:
- perplexity
base_model: facebook/opt-2.7b
---
## ptune-FLAN-OPT-2.7b
OPT was first introduced in [Open Pre-trained Transformer Language Models](https://arxiv.org/abs/2205.01068) and first released in [metaseq's repository](https://github.com/facebookresearch/metaseq) on May 3rd 2022 by Meta AI.
This model is [facebook/opt-2.7b](https://hf.co/facebook/opt-2.7b) finetuned with prefix tuning (https://arxiv.org/abs/2101.00190) on the FLAN datasets (https://arxiv.org/pdf/2210.11416.pdf).
A 24 token prefix was finetuned over 3.7m new tokens of a FLAN task mixture, with the start of each example cut off if it was too large to fit within a 512 token context.
The model reaches a train ppl of 5.95 and an eval ppl of 4.50.
### Example COT (Chain-of-thought) Prompt:
```
Q: Answer the following yes/no question by reasoning step-by-step. Could a dandelion suffer from hepatitis?
A: Hepatitis only affects organisms with livers. Dandelions don’t have a liver. The answer is no.
Q: Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?
A: A haiku is a japanese three-line poem. That is short enough to fit in 280 characters. The answer is yes.
Q: Answer the following yes/no question by reasoning step-by-step. Can you reach space with a Cessna?
A:
```
```
> Cessna is a type of aircraft. It has a wingspan of 3.6 meters (3.6m). The answer is no.
```
(Completed with Contrastive Sampling, top_k: 4, penalty_alpha: 0.6) |