Report for Sigma/financial-sentiment-analysis
#63
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 3 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_75agree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.393 | Transform to title case | 393/1000 tested samples (39.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 39.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2985 | It is a disappointment to see the plan folded . | It Is A Disappointment To See The Plan Folded . | LABEL_0 (p = 0.98) | LABEL_1 (p = 1.00) |
| 201 | Operating profit of Kauppalehti group rose to EUR 1.5 mn from EUR 1.3 mn , and that of Marketplaces to EUR 1.3 mn from EUR 1.0 mn in the third quarter of 2006 . | Operating Profit Of Kauppalehti Group Rose To Eur 1.5 Mn From Eur 1.3 Mn , And That Of Marketplaces To Eur 1.3 Mn From Eur 1.0 Mn In The Third Quarter Of 2006 . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| 3223 | Making matters more difficult , the company said it has been grappling with higher oil and gas prices , which have pushed up the cost of energy , raw materials and transportation . | Making Matters More Difficult , The Company Said It Has Been Grappling With Higher Oil And Gas Prices , Which Have Pushed Up The Cost Of Energy , Raw Materials And Transportation . | LABEL_0 (p = 1.00) | LABEL_1 (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.387 | Transform to uppercase | 387/1000 tested samples (38.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 38.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 202 | Operating profit rose from EUR 1.94 mn to EUR 2.45 mn . | OPERATING PROFIT ROSE FROM EUR 1.94 MN TO EUR 2.45 MN . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| 3223 | Making matters more difficult , the company said it has been grappling with higher oil and gas prices , which have pushed up the cost of energy , raw materials and transportation . | MAKING MATTERS MORE DIFFICULT , THE COMPANY SAID IT HAS BEEN GRAPPLING WITH HIGHER OIL AND GAS PRICES , WHICH HAVE PUSHED UP THE COST OF ENERGY , RAW MATERIALS AND TRANSPORTATION . | LABEL_0 (p = 1.00) | LABEL_1 (p = 1.00) |
| 1414 | Systeemitiimi 's sales and project resources will also be strengthened , director Paul Skogberg said . | SYSTEEMITIIMI 'S SALES AND PROJECT RESOURCES WILL ALSO BE STRENGTHENED , DIRECTOR PAUL SKOGBERG SAID . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.082 | Add typos | 82/1000 tested samples (8.2%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 8.2% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 551 | Finland 's Poyry Energy has won a contract to advise builders of a new cogeneration power plant in Lithuania 's second-biggest city of Kaunas and to supervise the construction process . | Finlabnd 's Poyry Enerygy has wkn a contract to advise bylider kf a new cogeneration power planti n Lithuania 's second-biggext city of Kaunas and to supervise the conztruction pfocess . | LABEL_2 (p = 1.00) | LABEL_1 (p = 0.99) |
| 1450 | `` We have analyzed Kaupthing Bank Sweden and found a business which fits well into Alandsbanken , '' said Alandsbanken 's chief executive Peter Wiklof in a statement . | `` We have naalyzed Kahupthing Ban Sweden and dound a business which fits well into Alandsbanken , '' said Alabndsbanken 's chkef executive Peter Wiklof in a statement . | LABEL_1 (p = 0.50) | LABEL_2 (p = 0.64) |
| 477 | narrows to EUR2 .8 m 9-mo '09 29 October 2009 - Finnish software and hardware developer Elektrobit Oyj HEL : EBG1V , or EB , said today that its net loss narrowed to EUR2 .8 m for the first nine months of 2009 from EUR35 .6 m for the same period a year ago . | narrows to EUR2 .8 m 9-mo '09 29 October 2009 - Finnish software and hardware developer Elektrobit Oyj HEL : EBG1V , or EB , said today that its net loss nzrrowed to EUR2 .8 m for the first nine mnths of 2009 from SUR35 .6 m for the same period a year ago . | LABEL_2 (p = 1.00) | LABEL_0 (p = 0.95) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!