qihoo360

/

360VL-8B

like 13

Instead of using the huggingface model names I created a local copy of the model using:

huggingface-cli download qihoo360/360VL-8B --local-dir ./360VL-8B

The provided example code will not work when supplying this local directory.
(I did not test it, but my assumption would be it also wouldn't work when using qihoo360/360VL-8B).

The reason is that the HoneybeeVisualProjectorConfig is not loaded correctly, meaning an error will happen when trying to apply the depth parameter in the CAbstractor class.

Taking a look at the code, I noticed that the config gets loaded as follows:

    if projector_type == 'c-abs':

        local_config_path = config.mm_projector_config
        honeybee_config = HoneybeeVisualProjectorConfig.from_pretrained(local_config_path)

So by default the same config file is used as the one which is supplied in this repository.

It seems as it is intended that this line then maps the correct config by using the visual_projector_config object of the main config.

        # get the visual_projector config dict if we are loading from HoneybeeConfig
        if config_dict.get("model_type") == "mllm":
            config_dict = config_dict["visual_projector_config"]

But the model type in the config is set to QH_360VL so this if branch is never run.

If I locally adjust this line to check for QH_360VL instead of mllm the code runs without a problem.