mp8/README.md

Addition from the Hugging Face team

This repository corresponds to the original Llama format and codebase, not the transformers library

The weights were originally distributed in the following format:

weights/
    consolidated.00.pth
    consolidated.01.pth
    ...

Unfortunately, the files themselves were too large for the Hub to handle, so we had to shard them. In order to keep the same structure that was originally given, the sharding is done as follows:

weights/
    consolidated.00/
        consolidated-00001-of-00011.pth
        consolidated-00002-of-00011.pth
        ...
    consolidated.01/
        consolidated-00001-of-00011.pth
        consolidated-00002-of-00011.pth
        ...
    ...

If trying to run the code that was given with the original weights, we recommend running this script to join the files together once again:

import os
import torch

from pathlib import Path

path_to_files = Path('.../weights')

folders = [folder for folder in os.listdir(path_to_files) if os.path.isdir(path_to_files / folder) and folder.startswith('consolidated')]
for folder in folders:
    state_dict = {}
    files = [file for file in  os.listdir(path_to_files / folder) if file.endswith(".pth")]

    for file in files:
        state_dict_partial = torch.load(path_to_files / folder / file, map_location="cpu")
        for key, value in state_dict_partial.items():
            state_dict[key]=value

    torch.save(state_dict, path_to_files / f"{folder}.pth")