datasets hackathon

BigScience🌸 Datasets Hackathon

Thank you for participating in the BigScience🌸 Datasets hackathon!

Setup

• Create an account at Hugging Face
• Request to be added to the BigScience organization

Install:

• Git
• Git LFS

How-to Guide

How to add a Collection

By default, collections are added as private community raw datasets in the 🤗 Hub, under the bigscience namespace.

1. Choose an unassigned open issue from Collections.

   The issues are sorted by priority depending on their license, size, among other criteria.

   In each Issue page, you can find detailed information of the collection, such as its identifier (UID) and location.

2. Self-assign you to that issue.

   In the Issue page, on the right column, under Assignees, click assign yourself.

3. Create a 🤗 Dataset repository: https://huggingface.co/new-dataset

   • Set Owner: bigscience
   • Set Dataset name: the collection identifier (UID)
   • Select Private
   • Create dataset

4. Clone the 🤗 Dataset repository:

   Replace <collection UID> with the collection identifier.

   git clone https://huggingface.co/datasets/bigscience/<collection UID>
   cd <collection UID>

5. Initialize Git LFS in the <collection UID> directory:

   cd <collection UID>
   git lfs install

6. Download the collection to the <collection UID> directory.

   Expected formats are:

   • TXT
   • JSON/JSONL
   • CSV
   • HTML/XML
   • WARC

   If you find another format, create an Issue (labeled as "data format") to decide whether/how to convert that format.

   • PDF: this format is hard to convert.

7. Compress the files, with gzip or zip.

8. Commit the files and push:

   git add .
   git commit -m "Add dataset"
   git push