Xenova HF staff commited on
Commit
9e75c88
1 Parent(s): 55c6b66

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +64 -1
README.md CHANGED
@@ -7,4 +7,67 @@ tags:
7
  - text2text-generation
8
  - image-text-to-text
9
  library_name: transformers.js
10
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  - text2text-generation
8
  - image-text-to-text
9
  library_name: transformers.js
10
+ ---
11
+
12
+ https://huggingface.co/microsoft/Florence-2-base with ONNX weights to be compatible with Transformers.js.
13
+
14
+ ## Usage (Transformers.js)
15
+
16
+ > [!IMPORTANT]
17
+ > NOTE: Florence-2 support is experimental and requires you to install Transformers.js [v3](https://github.com/xenova/transformers.js/tree/v3) from source.
18
+
19
+ If you haven't already, you can install the [Transformers.js](https://huggingface.co/docs/transformers.js) JavaScript library from [GitHub](https://github.com/xenova/transformers.js/tree/v3) using:
20
+ ```bash
21
+ npm install xenova/transformers.js#v3
22
+ ```
23
+
24
+ **Example:** Perform image captioning with `onnx-community/Florence-2-base`.
25
+ ```js
26
+ import {
27
+ Florence2ForConditionalGeneration,
28
+ AutoProcessor,
29
+ AutoTokenizer,
30
+ RawImage,
31
+ } from '@xenova/transformers';
32
+
33
+ // Load model, processor, and tokenizer
34
+ const model_id = 'onnx-community/Florence-2-base';
35
+ const model = await Florence2ForConditionalGeneration.from_pretrained(model_id, { dtype: 'fp32' });
36
+ const processor = await AutoProcessor.from_pretrained(model_id);
37
+ const tokenizer = await AutoTokenizer.from_pretrained(model_id);
38
+
39
+ // Load image and prepare vision inputs
40
+ const url = 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg';
41
+ const image = await RawImage.fromURL(url);
42
+ const vision_inputs = await processor(image);
43
+
44
+ // Specify task and prepare text inputs
45
+ const task = '<MORE_DETAILED_CAPTION>';
46
+ const prompts = processor.construct_prompts(task);
47
+ const text_inputs = tokenizer(prompts);
48
+
49
+ // Generate text
50
+ const generated_ids = await model.generate({
51
+ ...text_inputs,
52
+ ...vision_inputs,
53
+ max_new_tokens: 100,
54
+ });
55
+
56
+ // Decode generated text
57
+ const generated_text = tokenizer.batch_decode(generated_ids, { skip_special_tokens: false })[0];
58
+
59
+ // Post-process the generated text
60
+ const result = processor.post_process_generation(generated_text, task, image.size);
61
+ console.log(result);
62
+ // { '<MORE_DETAILED_CAPTION>': 'The image shows a vintage Volkswagen Beetle car parked on a cobblestone street in front of a yellow building with two wooden doors. The car is a light green color with silver rims and appears to be in good condition. The building has a sloping roof and is painted in a combination of yellow and beige colors. The sky is blue and there are trees in the background. The overall mood of the image is peaceful and serene.' }
63
+ ```
64
+
65
+ We also released an online demo, which you can try yourself: https://huggingface.co/spaces/Xenova/florence2-webgpu
66
+
67
+
68
+ <video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/61b253b7ac5ecaae3d1efe0c/BJj3jQXNqS_7Nt2MSb2ss.mp4"></video>
69
+
70
+ ---
71
+
72
+ Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using [🤗 Optimum](https://huggingface.co/docs/optimum/index) and structuring your repo like this one (with ONNX weights located in a subfolder named `onnx`).
73
+