datasets hackathon

BigScience🌸 Datasets Hackathon

Thank you for participating in the BigScience🌸 Datasets hackathon!

Setup

• Create an account at Hugging Face
• Request to join the BigScience Catalogue Data organization

Install:

• Git
• Git LFS

How-to Guide

How to add a Collection

By default, collections are added as private community raw datasets in the 🤗 Hub, under the bigscience namespace.

1. Choose an unassigned open issue from Collections.

   The issues are sorted by priority depending on their license, size, among other criteria.

   In each Issue page, you can find detailed information of the collection, such as its identifier (UID) and location.

2. Self-assign you to that issue.

   In the Issue page, make a comment containing only the keyword #self-assign.

3. Create a 🤗 Dataset repository: https://huggingface.co/new-dataset

   • Set Owner: bigscience
   • Set Dataset name: the collection identifier (UID)
   • Select Private
   • Create dataset

4. Clone the 🤗 Dataset repository:

   Replace <collection UID> with the collection identifier.

   git clone https://huggingface.co/datasets/bigscience/<collection UID>

5. Initialize Git LFS in the <collection UID> directory:

   cd <collection UID>
   git lfs install

6. Download the collection to the <collection UID> directory.

   Expected formats are:

   • TXT
   • JSON/JSONL
   • CSV
   • HTML/XML
   • WARC

   If you find another format:

   • Search (or create) an Issue labeled as "data format" to decide whether/how to convert that format. That will be addressed as an optional subsequent step.

   • Make a comment in the "data catalog" collection issue referring the "data format" issue number:

     Replace <"data format" issue number> with the corresponding "data format" issue number:

     This dataset format needs being converted:
     - #<"data format" issue number>
     

   Other formats you may find:

   • PDF

7. Compress the files, with gzip or zip.

8. Commit the files and push:

   git add .
   git commit -m "Add dataset"
   git push