Skip to content

Use nix instead of json in output of hackage-to-nix #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 3, 2022

Conversation

hamishmack
Copy link
Contributor

@hamishmack hamishmack commented Aug 3, 2022

We have been using readFile and fromJSON to read a .json a 60MB .json file. This change puts the information contained in the json file into .nix files that are imported when needed (rather than all at once). This seems to save around 1s at eval time.

Tested with:

time nix-instantiate -E '(import ./. {}).pkgs-unstable.haskell-nix.tool "ghc8107" "hello" {}'
time nix-instantiate -E '(import ./. {}).pkgs-unstable.haskell-nix.tool "ghc8107" "haskell-language-server" {}'

See also input-output-hk/haskell.nix#1574

@hamishmack hamishmack requested a review from angerman August 3, 2022 10:15

-- Avoid issues with case insensitive file systems by escaping upper case
-- characters with a leading _ character.
escapeUpperCase :: String -> String
Copy link
Contributor

@yvan-sraka yvan-sraka Aug 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds a bit like a custom hack to me … what about just add something like a base64 representation of strings we want to encode?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is for filenames, base64 encoding would make it a lot less readable :-/

I wonder if we'd run into camelCase issues.

That would be converted into camel_case, and thus clash with a package named camel_case? Can we rule out that this would happen?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted the file names to still be human readable. I guess we could include both human and base64 encoding, but then it is kind of a custom format again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My hot take would be that here we had a good opportunity to hide this hack behind an abstraction, meaning a small lib. This is a micro Haskell project I would be glad to publish on Hackage!

Copy link
Collaborator

@angerman angerman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we extend the commit message with the motivation and performance observations?


-- Avoid issues with case insensitive file systems by escaping upper case
-- characters with a leading _ character.
escapeUpperCase :: String -> String
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is for filenames, base64 encoding would make it a lot less readable :-/

I wonder if we'd run into camelCase issues.

That would be converted into camel_case, and thus clash with a package named camel_case? Can we rule out that this would happen?

@hamishmack
Copy link
Contributor Author

hamishmack commented Aug 3, 2022

That would be converted into camel_case, and thus clash with a package named camel_case? Can we rule out that this would happen?

It replaces _ with __ to prevent that. So:

camelCase -> camel_Case
camel_case -> camel__case

@hamishmack hamishmack merged commit 555d57e into master Aug 3, 2022
hamishmack added a commit to input-output-hk/haskell.nix that referenced this pull request Aug 4, 2022
We have been using `readFile` and `fromJSON` to read a `.json` a 60MB .json file.  This change puts the information contained in the json file into `.nix` files that are imported when needed (rather than all at once).  This seems to save around 1s at eval time.

Tested with:
```
time nix-instantiate -E '(import ./. {}).pkgs-unstable.haskell-nix.tool "ghc8107" "hello" {}'
time nix-instantiate -E '(import ./. {}).pkgs-unstable.haskell-nix.tool "ghc8107" "haskell-language-server" {}'
```

See also input-output-hk/nix-tools#118
@yvan-sraka yvan-sraka deleted the hkm/nixify-hackage-to-nix branch September 15, 2022 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants