Skip to content

Commit 5b06c2d

Browse files
nik9000nik9000
authored
Allow building all docs with a local repo (#659)
We current offer two ways to build the docs: 1. Build an arbitrary book with `--doc` and its companion arguments like `--resource` and `--asciidoctor`. 2. Build *all* of the books that Elastic publishes with `--all`. Building an arbitrary book is nice and quick because it is have very little overhead on top of the time it takes to build the actual book. But it isn't always the best test because the command that you use to build the book might not exactly match the invocation that `--all` uses to build the book. Furthermore, building a single book is not usually enough! When you are testing documentation changes that you've made locally you often don't know all of the books that your changes might effect. The Elasticsearch repository is used in about a dozen books in combination with a hand full of other repositories. Building all of the docs is quite slow and you might run into errors that you didn't cause. It is exactly what we need to build the docs for the public web site but it isn't right for testing some changes that you make locally. It also doesn't line up properly with local development as it only knows how to pull repos from the Elastic organization at github. This change adds a "third way" to build test the docs that splits the difference between the two options. It uses the same mechanisms that we usually use to build all of the docs but allows you to substitute local directories in place of some branch of some repo: ``` ./build_docs --all --target_repo [email protected]:elastic/built-docs.git \ --keep_hash \ --sub_dir elasticsearch:master:~/Code/elasticsearch ``` This is nice because it: 1. Rebuilds all of the books that your local directory is involved with 2. Uses standard switches when building each of those books 3. Does not rebuild books that don't use your local directory That third thing is actually caused by that new `--keep_hash` flag. It causes the build's up-to-date checking to use the same hash that was used the last time the book was built. `--keep_hash` has uses beyond being paired with `--sub_dir`. In particular, it is useful when testing what happens when you switch a book to Asciidoctor. You can switch the book to Asciidoctor in `conf.yaml` and rebuild with `--keep_hash` and the build will *only* have the asciidoctor change in it. This also drop `--no_fetch` because `--keep_hash` is better in almost every way. The goal of `--sub_dir` isn't so much that people will execute it locally, but that we can execute it on CI. I'm sure folks will execute it locally but it pretty heavy because, just like a normal `--all` build it has to: 1. Clone the `built-docs` repo or fetch updates if it is already cloned. 2. Check out the `built-docs` repo. This is truly slow because it has many, many files in it. 3. Clone all of the repos used to build every book or fetch updates if they are already cloned. I suspect in time we can fix some of these. But for now, this is a heavy thing.
1 parent 3b6eaf0 commit 5b06c2d

File tree

9 files changed

+365
-85
lines changed

9 files changed

+365
-85
lines changed

‎build_docs

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,8 @@ from __future__ import print_function
2929

3030
import logging
3131
from os import environ, getgid, getuid
32-
from os.path import basename, dirname, exists, isdir, join, realpath
32+
from os.path import basename, dirname, exists, expanduser, isdir
33+
from os.path import join, realpath
3334
import re
3435
import subprocess
3536
from sys import platform, version_info
@@ -117,8 +118,7 @@ def run_build_docs(args):
117118
'-v',
118119
'%s:%s:ro,cached' % (repo_root, repo_mount)
119120
])
120-
build_docs_args.append(
121-
repo_mount + path.replace(repo_root, ''))
121+
return repo_mount + path.replace(repo_root, '')
122122

123123
open_browser = False
124124
args = Args(args)
@@ -164,7 +164,8 @@ def run_build_docs(args):
164164
doc_file = realpath(args.next_arg_or_err())
165165
if not exists(doc_file):
166166
raise ArgError("Can't find --doc %s" % doc_file)
167-
mount_docs_repo_and_dockerify_path(dirname(doc_file), doc_file)
167+
build_docs_args.append(mount_docs_repo_and_dockerify_path(
168+
dirname(doc_file), doc_file))
168169
saw_doc = True
169170
elif arg == '--open':
170171
docker_args.extend(['--publish', '8000:8000/tcp'])
@@ -200,7 +201,19 @@ def run_build_docs(args):
200201
resource_dir = realpath(args.next_arg_or_err())
201202
if not isdir(resource_dir):
202203
raise ArgError("Can't find --resource %s" % resource_dir)
203-
mount_docs_repo_and_dockerify_path(resource_dir, resource_dir)
204+
build_docs_args.append(mount_docs_repo_and_dockerify_path(
205+
resource_dir, resource_dir))
206+
elif arg == '--sub_dir':
207+
sub = args.next_arg_or_err()
208+
m = re.match('(?P<repo>[^:]+):(?P<branch>[^:]+):(?P<dir>.+)', sub)
209+
if not m:
210+
raise ArgError("Invalid --sub_dir %s" % sub)
211+
sub_dir = realpath(expanduser(m.group('dir')))
212+
if not exists(sub_dir):
213+
raise ArgError("Can't find --sub_dir %s" % sub_dir)
214+
mounted_path = mount_docs_repo_and_dockerify_path(sub_dir, sub_dir)
215+
build_docs_args.append("%s:%s:%s" % (
216+
m.group('repo'), m.group('branch'), mounted_path))
204217
arg = args.next_arg()
205218

206219
if saw_doc and not saw_out:

‎build_docs.pl

Lines changed: 26 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ BEGIN
5555

5656
GetOptions(
5757
$Opts, #
58-
'all', 'push', 'target_repo=s', 'reference=s', 'rebuild', 'no_fetch', #
58+
'all', 'push', 'target_repo=s', 'reference=s', 'rebuild', 'keep_hash', 'sub_dir=s@',
5959
'single', 'pdf', 'doc=s', 'out=s', 'toc', 'chunk=i', 'suppress_migration_warnings',
6060
'open', 'skiplinkcheck', 'linkcheckonly', 'staging', 'procs=i', 'user=s', 'lang=s',
6161
'lenient', 'verbose', 'reload_template', 'resource=s@', 'asciidoctor', 'in_standard_docker',
@@ -464,7 +464,11 @@ sub init_repos {
464464
user => $Opts->{user},
465465
url => $Opts->{target_repo},
466466
reference => $reference_dir,
467-
# intentionally not passing the tracker because we don't want to use it
467+
# We can't keep the hash of the target repo because it is what stores
468+
# the hashes in the first place!
469+
keep_hash => 0,
470+
# Intentionally not passing the tracker because we need to build the
471+
# tracker from information in this repo.
468472
);
469473
delete $child_dirs{ $target_repo->git_dir->absolute };
470474
my $target_repo_checkout = "$temp_dir/target_repo";
@@ -498,6 +502,7 @@ sub init_repos {
498502
user => $Opts->{user},
499503
url => $url,
500504
reference => $reference_dir,
505+
keep_hash => $Opts->{keep_hash},
501506
);
502507
delete $child_dirs{ $repo->git_dir->absolute };
503508

@@ -507,7 +512,7 @@ sub init_repos {
507512
else {
508513
$pm->start($name) and next;
509514
eval {
510-
$repo->update_from_remote() unless $Opts->{no_fetch};
515+
$repo->update_from_remote();
511516
1;
512517
} or do {
513518
# If creds are invalid, explicitly reject them to try to clear the cache
@@ -522,6 +527,16 @@ sub init_repos {
522527
}
523528
$pm->wait_all_children;
524529

530+
# Parse the --sub_dir options and attach the to the repo
531+
my %sub_dirs = ();
532+
foreach (@{ $Opts->{sub_dir} }) {
533+
die "invalid --sub_dir $_"
534+
unless /(?<repo>[^:]+):(?<branch>[^:]+):(?<dir>.+)/;
535+
my $dir = dir($+{dir})->absolute;
536+
die "--sub_dir $dir doesn't exist" unless -e $dir;
537+
ES::Repo->get_repo($+{repo})->add_sub_dir($+{branch}, $dir);
538+
}
539+
525540
for ( keys %child_dirs ) {
526541
my $dir = dir($_);
527542
next unless -d $dir;
@@ -682,7 +697,8 @@ sub check_args {
682697
die('--user not compatible with --doc') if $Opts->{user};
683698
die('--reference not compatible with --doc') if $Opts->{reference};
684699
die('--rebuild not compatible with --doc') if $Opts->{rebuild};
685-
die('--no_fetch not compatible with --doc') if $Opts->{no_fetch};
700+
die('--keep_hash not compatible with --doc') if $Opts->{keep_hash};
701+
die('--sub_dir not compatible with --doc') if $Opts->{sub_dir};
686702
die('--skiplinkcheck not compatible with --doc') if $Opts->{skiplinkcheck};
687703
die('--linkcheckonly not compatible with --doc') if $Opts->{linkcheckonly};
688704
} else {
@@ -703,9 +719,10 @@ sub pick_conf {
703719
#===================================
704720
return 'conf.yaml' unless $Opts->{conf};
705721

706-
my $conf = dir($Old_Pwd)->file($Opts->{conf});
722+
my $conf = file($Opts->{conf});
723+
$conf = dir($Old_Pwd)->file($Opts->{conf}) if $conf->is_relative;
707724
return $conf if -e $conf;
708-
die $Opts->{conf} . " doesn't exist";
725+
die "$conf doesn't exist";
709726
}
710727

711728
#===================================
@@ -793,7 +810,9 @@ sub usage {
793810
--skiplinkcheck Omit the step that checks for broken links
794811
--linkcheckonly Skips the documentation builds. Checks links only.
795812
--rebuild Rebuild all branches of every book regardless of what has changed
796-
--no_fetch Skip fetching updates from source repos
813+
--keep_hash Build docs from the same commit hash as last time
814+
--sub_dir Use a directory as a branch of some repo
815+
(eg --sub_dir elasticsearch:master:~/Code/elasticsearch)
797816
798817
General Opts:
799818
--staging Use the template from the staging website

‎integtest/Makefile

Lines changed: 160 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
SHELL = /bin/bash -eux -o pipefail
22
MAKEFLAGS += --silent
3+
TMP = /tmp/docs_integtest/$@
34

45
# Used by the test for --all
56
export GIT_AUTHOR_NAME=Test
@@ -18,7 +19,11 @@ check: \
1819
missing_include_fails_asciidoc missing_include_fails_asciidoctor \
1920
migration_warnings \
2021
readme_expected_files readme_same_files \
21-
small_all_expected_files
22+
simple_all \
23+
relative_conf_file \
24+
keep_hash \
25+
sub_dir \
26+
keep_hash_and_sub_dir
2227

2328
.PHONY: style
2429
style: html_diff
@@ -122,38 +127,170 @@ migration_warnings: migration_warnings.asciidoc
122127
/tmp/%_asciidoctor:
123128
$(BD) --asciidoctor --doc $*.asciidoc
124129

125-
.PHONY: small_all_expected_files
126-
small_all_expected_files: /tmp/small_all
127-
[ -s $^/redirects.conf ]
128-
[ -s $^/html/branches.yaml ]
129-
grep '<a class="ulink" href="test/current/index.html" target="_top">Test book</a>' $^/html/index.html > /dev/null
130-
grep '<meta http-equiv="refresh" content="0; url=current/index.html">' $^/html/test/index.html > /dev/null
131-
[ -s $^/html/test/current/index.html ]
130+
.PHONY: simple_all
131+
simple_all:
132+
# Test the simplest possible `--all` invocation
133+
rm -rf $(TMP)
134+
$(BUILD_MINIMAL_ALL)
135+
$(MINIMAL_ALL_EXPECTED_FILES)
132136

133-
.PRECIOUS: /tmp/small_all
134-
/tmp/small_all:
135-
# Builds "--all" documentation specified by by the "small_conf.yaml" file.
137+
.PHONY: relative_conf_file
138+
relative_conf_file:
139+
# Make sure that using a relative referece to the --conf file works.
140+
rm -rf $(TMP)
141+
$(SETUP_MINIMAL_ALL)
142+
cd $(TMP) && \
143+
/docs_build/build_docs.pl --in_standard_docker --all --push \
144+
--target_repo $(TMP)/dest.git \
145+
--conf conf.yaml
146+
$(MINIMAL_ALL_EXPECTED_FILES)
136147

137-
# First build a repository to use as the source.
138-
rm -rf /tmp/source
139-
git init /tmp/source
140-
cp minimal.asciidoc /tmp/source/
141-
cd /tmp/source && \
148+
.PHONY: keep_hash
149+
keep_hash:
150+
# Test that `--all --keep_hash` doesn't pull new updates
151+
rm -rf $(TMP)
152+
$(BUILD_MINIMAL_ALL)
153+
154+
# REPLACE the minimal documentation in the source repo
155+
cp ../README.asciidoc $(TMP)/source/index.asciidoc
156+
cd $(TMP)/source && \
157+
git add . && \
158+
git commit -m 'README'
159+
160+
# Rebuild the docs with --keep_hash which should ignore the replacement
161+
/docs_build/build_docs.pl --in_standard_docker --all --push \
162+
--target_repo $(TMP)/dest.git \
163+
--conf $(TMP)/conf.yaml \
164+
--keep_hash | tee $(TMP)/out
165+
$(call GREP,'No changes to push',$(TMP)/out)
166+
167+
# We expact the same files as the minimal because we the changes that we
168+
# make shouldn't be included
169+
$(MINIMAL_ALL_EXPECTED_FILES)
170+
171+
.PHONY: sub_dir
172+
sub_dir:
173+
# Test that `--all --sub_dir` substitutes a directory for a branch of
174+
# a repo.
175+
rm -rf $(TMP)
176+
177+
# We still need to build the source repo because the script wants to fetch
178+
# it just in case we need another branch.
179+
git init $(TMP)/source
180+
cd $(TMP)/source && git commit --allow-empty -m "empty"
181+
182+
# Setup the directory we'd like to substitute
183+
mkdir $(TMP)/to_sub
184+
cp minimal.asciidoc $(TMP)/to_sub/index.asciidoc
185+
186+
git init --bare $(TMP)/dest.git
187+
sed -e 's|--tmp--|$(TMP)|' small_conf.yaml > $(TMP)/conf.yaml
188+
/docs_build/build_docs.pl --in_standard_docker --all --push \
189+
--target_repo $(TMP)/dest.git \
190+
--conf $(TMP)/conf.yaml \
191+
--sub_dir source:master:$(TMP)/to_sub
192+
193+
$(MINIMAL_ALL_EXPECTED_FILES)
194+
195+
.PHONY: keep_hash_and_sub_dir
196+
keep_hash_and_sub_dir:
197+
# Test that `--all --keep_hash --sub_dir` keeps hashes the same for repos
198+
# not specified by --sub_dir but forces rebuilding all books that include
199+
# --sub_dir.
200+
201+
rm -rf $(TMP)
202+
$(call INIT_REPO_WITH_FILE,$(TMP)/source1,includes_source2.asciidoc,docs/index.asciidoc)
203+
$(call INIT_REPO_WITH_FILE,$(TMP)/source2,included.asciidoc,index.asciidoc)
204+
git init --bare $(TMP)/dest.git
205+
206+
sed 's|--tmp--|$(TMP)|' two_repos_conf.yaml > $(TMP)/conf.yaml
207+
/docs_build/build_docs.pl --in_standard_docker --all --push \
208+
--target_repo $(TMP)/dest.git \
209+
--conf $(TMP)/conf.yaml
210+
211+
# Move a "bad" file into source2 so we can be sure we're not picking it up
212+
cp ../README.asciidoc $(TMP)/source2/index.asciidoc
213+
cd $(TMP)/source2 && \
142214
git add . && \
143-
git commit -m 'minimal'
215+
git commit -m 'README'
216+
217+
/docs_build/build_docs.pl --in_standard_docker --all --push \
218+
--target_repo $(TMP)/dest.git \
219+
--conf $(TMP)/conf.yaml \
220+
--keep_hash | tee /tmp/out
221+
$(call GREP,'No changes to push',/tmp/out)
222+
223+
# Setup the directory we'd like to substitute
224+
mkdir -p $(TMP)/to_sub/docs
225+
cp includes_source2.asciidoc $(TMP)/to_sub/docs/index.asciidoc
226+
echo "extra extra extra" >> $(TMP)/to_sub/docs/index.asciidoc
227+
228+
/docs_build/build_docs.pl --in_standard_docker --all --push \
229+
--target_repo $(TMP)/dest.git \
230+
--conf $(TMP)/conf.yaml \
231+
--keep_hash --sub_dir source1:master:$(TMP)/to_sub | tee $(TMP)/out
232+
$(call GREP,'Pushing changes',$(TMP)/out)
233+
234+
git clone $(TMP)/dest.git $(TMP)/dest
235+
$(call GREP,'extra extra extra',$(TMP)/dest/html/test/current/_chapter.html)
236+
237+
define GREP=
238+
# grep for a string in a file, outputting the whole file if there isn't
239+
# a match.
240+
[ -e $(2) ] || { \
241+
echo "can't find $(2)"; \
242+
ls $$(dirname $(2)); \
243+
false; \
244+
}
245+
grep $(1) $(2) > /dev/null || { \
246+
echo "Couldn't find $(1) in $(2):"; \
247+
cat $(2); \
248+
false; \
249+
}
250+
endef
251+
252+
define SETUP_MINIMAL_ALL=
253+
# First build a repository to use as the source.
254+
$(call INIT_REPO_WITH_FILE,$(TMP)/source,minimal.asciidoc,index.asciidoc)
144255

145256
# Initialize a bare repository that the docs build process can use as a
146257
# remote. It is used to pushing to github but it can push to a remote on
147258
# the filesystem just fine.
148-
git init --bare /tmp/small_all.git
259+
git init --bare $(TMP)/dest.git
149260

150261
# Actually build the docs
262+
sed 's|--tmp--|$(TMP)|' small_conf.yaml > $(TMP)/conf.yaml
263+
endef
264+
265+
define BUILD_MINIMAL_ALL=
266+
# Builds `--all` docs using a "minimal" source file
267+
$(SETUP_MINIMAL_ALL)
151268
/docs_build/build_docs.pl --in_standard_docker --all --push \
152-
--target_repo /tmp/small_all.git \
153-
--conf small_conf.yaml
269+
--target_repo $(TMP)/dest.git \
270+
--conf $(TMP)/conf.yaml
271+
endef
154272

155-
# Check out the files we just built
156-
git clone /tmp/small_all.git /tmp/small_all
273+
define MINIMAL_ALL_EXPECTED_FILES=
274+
# Checks that $(1).git contains the expected result of building the
275+
# "minimal" source file
276+
git clone $(TMP)/dest.git $(TMP)/dest
277+
[ -s $(TMP)/dest/redirects.conf ]
278+
[ -s $(TMP)/dest/html/branches.yaml ]
279+
$(call GREP,'<a class="ulink" href="test/current/index.html" target="_top">Test book</a>',$(TMP)/dest/html/index.html)
280+
$(call GREP,'<meta http-equiv="refresh" content="0; url=current/index.html">',$(TMP)/dest/html/test/index.html)
281+
[ -s $(TMP)/dest/html/test/current/index.html ]
282+
endef
283+
284+
define INIT_REPO_WITH_FILE=
285+
# Initializes the repo at $(1) and commits the file in $(2) with the
286+
# name $(3)
287+
git init $(1)
288+
mkdir -p $$(dirname $(1)/$(3))
289+
cp $(2) $(1)/$(3)
290+
cd $(1) && \
291+
git add . && \
292+
git commit -m 'init'
293+
endef
157294

158295
define GREP=
159296
# grep for a string in a file, outputting the whole file if there isn't
@@ -168,4 +305,4 @@ define GREP=
168305
cat $(2); \
169306
false; \
170307
}
171-
endef
308+
endef

‎integtest/includes_source2.asciidoc

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
= Title
2+
3+
== Chapter
4+
5+
I include simple between here
6+
7+
include::../../source2/index.asciidoc[]
8+
9+
and here.

0 commit comments

Comments
 (0)