Skip to content

Commit 4c0f47c

Browse files
jeffhostetlerdscho
jeffhostetler
authored and
dscho
committed
Merge pull request #3082 from dscho/fsmonitor-gfw
Add an experimental built-in FSMonitor
2 parents 461d4bb + 7c5b101 commit 4c0f47c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+7158
-77
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@
7171
/git-format-patch
7272
/git-fsck
7373
/git-fsck-objects
74+
/git-fsmonitor--daemon
7475
/git-gc
7576
/git-get-tar-commit-id
7677
/git-grep

‎Documentation/config/core.txt

Lines changed: 37 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -66,18 +66,45 @@ core.fsmonitor::
6666
will identify all files that may have changed since the
6767
requested date/time. This information is used to speed up git by
6868
avoiding unnecessary processing of files that have not changed.
69-
See the "fsmonitor-watchman" section of linkgit:githooks[5].
69+
+
70+
See the "fsmonitor-watchman" section of linkgit:githooks[5].
71+
+
72+
Note: FSMonitor hooks (and this config setting) are ignored if the
73+
(experimental) built-in FSMonitor is enabled (see
74+
`core.useBuiltinFSMonitor`).
7075

7176
core.fsmonitorHookVersion::
72-
Sets the version of hook that is to be used when calling fsmonitor.
73-
There are currently versions 1 and 2. When this is not set,
74-
version 2 will be tried first and if it fails then version 1
75-
will be tried. Version 1 uses a timestamp as input to determine
76-
which files have changes since that time but some monitors
77-
like watchman have race conditions when used with a timestamp.
78-
Version 2 uses an opaque string so that the monitor can return
79-
something that can be used to determine what files have changed
80-
without race conditions.
77+
Sets the version of hook that is to be used when calling the
78+
FSMonitor hook (as configured via `core.fsmonitor`).
79+
+
80+
There are currently versions 1 and 2. When this is not set,
81+
version 2 will be tried first and if it fails then version 1
82+
will be tried. Version 1 uses a timestamp as input to determine
83+
which files have changes since that time but some monitors
84+
like watchman have race conditions when used with a timestamp.
85+
Version 2 uses an opaque string so that the monitor can return
86+
something that can be used to determine what files have changed
87+
without race conditions.
88+
+
89+
Note: FSMonitor hooks (and this config setting) are ignored if the
90+
built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).
91+
92+
core.useBuiltinFSMonitor::
93+
(EXPERIMENTAL) If set to true, enable the built-in filesystem
94+
event watcher (for technical details, see
95+
linkgit:git-fsmonitor--daemon[1]).
96+
+
97+
Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up
98+
Git commands that need to refresh the Git index (e.g. `git status`) in a
99+
worktree with many files. The built-in FSMonitor facility eliminates the
100+
need to install and maintain an external third-party monitoring tool.
101+
+
102+
The built-in FSMonitor is currently available only on a limited set of
103+
supported platforms.
104+
+
105+
Note: if this config setting is set to `true`, any FSMonitor hook
106+
configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`)
107+
is ignored.
81108

82109
core.trustctime::
83110
If false, the ctime differences between the index and the

‎Documentation/git-fsmonitor--daemon.txt

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
git-fsmonitor--daemon(1)
2+
========================
3+
4+
NAME
5+
----
6+
git-fsmonitor--daemon - (EXPERIMENTAL) Builtin file system monitor daemon
7+
8+
SYNOPSIS
9+
--------
10+
[verse]
11+
'git fsmonitor--daemon' --start
12+
'git fsmonitor--daemon' --run
13+
'git fsmonitor--daemon' --stop
14+
'git fsmonitor--daemon' --is-running
15+
'git fsmonitor--daemon' --is-supported
16+
'git fsmonitor--daemon' --query <token>
17+
'git fsmonitor--daemon' --query-index
18+
'git fsmonitor--daemon' --flush
19+
20+
DESCRIPTION
21+
-----------
22+
23+
NOTE! This command is still only an experiment, subject to change dramatically
24+
(or even to be abandoned).
25+
26+
Monitors files and directories in the working directory for changes using
27+
platform-specific file system notification facilities.
28+
29+
It communicates directly with commands like `git status` using the
30+
link:technical/api-simple-ipc.html[simple IPC] interface instead of
31+
the slower linkgit:githooks[5] interface.
32+
33+
OPTIONS
34+
-------
35+
36+
--start::
37+
Starts the fsmonitor daemon in the background.
38+
39+
--run::
40+
Runs the fsmonitor daemon in the foreground.
41+
42+
--stop::
43+
Stops the fsmonitor daemon running for the current working
44+
directory, if present.
45+
46+
--is-running::
47+
Exits with zero status if the fsmonitor daemon is watching the
48+
current working directory.
49+
50+
--is-supported::
51+
Exits with zero status if the fsmonitor daemon feature is supported
52+
on this platform.
53+
54+
--query <token>::
55+
Connects to the fsmonitor daemon (starting it if necessary) and
56+
requests the list of changed files and directories since the
57+
given token.
58+
This is intended for testing purposes.
59+
60+
--query-index::
61+
Read the current `<token>` from the File System Monitor index
62+
extension (if present) and use it to query the fsmonitor daemon.
63+
This is intended for testing purposes.
64+
65+
--flush::
66+
Force the fsmonitor daemon to flush its in-memory cache and
67+
re-sync with the file system.
68+
This is intended for testing purposes.
69+
70+
REMARKS
71+
-------
72+
The fsmonitor daemon is a long running process that will watch a single
73+
working directory. Commands, such as `git status`, should automatically
74+
start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true`
75+
(see linkgit:git-config[1]).
76+
77+
Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each
78+
working directory separately, or globally via `git config --global
79+
core.useBuiltinFSMonitor true`.
80+
81+
Tokens are opaque strings. They are used by the fsmonitor daemon to
82+
mark a point in time and the associated internal state. Callers should
83+
make no assumptions about the content of the token. In particular,
84+
the should not assume that it is a timestamp.
85+
86+
Query commands send a request-token to the daemon and it responds with
87+
a summary of the changes that have occurred since that token was
88+
created. The daemon also returns a response-token that the client can
89+
use in a future query.
90+
91+
For more information see the "File System Monitor" section in
92+
linkgit:git-update-index[1].
93+
94+
CAVEATS
95+
-------
96+
97+
The fsmonitor daemon does not currently know about submodules and does
98+
not know to filter out file system events that happen within a
99+
submodule. If fsmonitor daemon is watching a super repo and a file is
100+
modified within the working directory of a submodule, it will report
101+
the change (as happening against the super repo). However, the client
102+
should properly ignore these extra events, so performance may be affected
103+
but it should not cause an incorrect result.
104+
105+
GIT
106+
---
107+
Part of the linkgit:git[1] suite

‎Documentation/git-update-index.txt

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR
498498
This feature is intended to speed up git operations for repos that have
499499
large working directories.
500500

501-
It enables git to work together with a file system monitor (see the
501+
It enables git to work together with a file system monitor (see
502+
linkgit:git-fsmonitor--daemon[1]
503+
and the
502504
"fsmonitor-watchman" section of linkgit:githooks[5]) that can
503505
inform it as to what files have been modified. This enables git to avoid
504506
having to lstat() every file to find modified files.

‎Documentation/githooks.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -584,7 +584,8 @@ fsmonitor-watchman
584584

585585
This hook is invoked when the configuration option `core.fsmonitor` is
586586
set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`
587-
depending on the version of the hook to use.
587+
depending on the version of the hook to use, unless overridden via
588+
`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).
588589

589590
Version 1 takes two arguments, a version (1) and the time in elapsed
590591
nanoseconds since midnight, January 1, 1970.

‎Documentation/technical/api-simple-ipc.txt

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
Simple-IPC API
2+
==============
3+
4+
The Simple-IPC API is a collection of `ipc_` prefixed library routines
5+
and a basic communication protocol that allow an IPC-client process to
6+
send an application-specific IPC-request message to an IPC-server
7+
process and receive an application-specific IPC-response message.
8+
9+
Communication occurs over a named pipe on Windows and a Unix domain
10+
socket on other platforms. IPC-clients and IPC-servers rendezvous at
11+
a previously agreed-to application-specific pathname (which is outside
12+
the scope of this design) that is local to the computer system.
13+
14+
The IPC-server routines within the server application process create a
15+
thread pool to listen for connections and receive request messages
16+
from multiple concurrent IPC-clients. When received, these messages
17+
are dispatched up to the server application callbacks for handling.
18+
IPC-server routines then incrementally relay responses back to the
19+
IPC-client.
20+
21+
The IPC-client routines within a client application process connect
22+
to the IPC-server and send a request message and wait for a response.
23+
When received, the response is returned back the caller.
24+
25+
For example, the `fsmonitor--daemon` feature will be built as a server
26+
application on top of the IPC-server library routines. It will have
27+
threads watching for file system events and a thread pool waiting for
28+
client connections. Clients, such as `git status` will request a list
29+
of file system events since a point in time and the server will
30+
respond with a list of changed files and directories. The formats of
31+
the request and response are application-specific; the IPC-client and
32+
IPC-server routines treat them as opaque byte streams.
33+
34+
35+
Comparison with sub-process model
36+
---------------------------------
37+
38+
The Simple-IPC mechanism differs from the existing `sub-process.c`
39+
model (Documentation/technical/long-running-process-protocol.txt) and
40+
used by applications like Git-LFS. In the LFS-style sub-process model
41+
the helper is started by the foreground process, communication happens
42+
via a pair of file descriptors bound to the stdin/stdout of the
43+
sub-process, the sub-process only serves the current foreground
44+
process, and the sub-process exits when the foreground process
45+
terminates.
46+
47+
In the Simple-IPC model the server is a very long-running service. It
48+
can service many clients at the same time and has a private socket or
49+
named pipe connection to each active client. It might be started
50+
(on-demand) by the current client process or it might have been
51+
started by a previous client or by the OS at boot time. The server
52+
process is not associated with a terminal and it persists after
53+
clients terminate. Clients do not have access to the stdin/stdout of
54+
the server process and therefore must communicate over sockets or
55+
named pipes.
56+
57+
58+
Server startup and shutdown
59+
---------------------------
60+
61+
How an application server based upon IPC-server is started is also
62+
outside the scope of the Simple-IPC design and is a property of the
63+
application using it. For example, the server might be started or
64+
restarted during routine maintenance operations, or it might be
65+
started as a system service during the system boot-up sequence, or it
66+
might be started on-demand by a foreground Git command when needed.
67+
68+
Similarly, server shutdown is a property of the application using
69+
the simple-ipc routines. For example, the server might decide to
70+
shutdown when idle or only upon explicit request.
71+
72+
73+
Simple-IPC protocol
74+
-------------------
75+
76+
The Simple-IPC protocol consists of a single request message from the
77+
client and an optional response message from the server. Both the
78+
client and server messages are unlimited in length and are terminated
79+
with a flush packet.
80+
81+
The pkt-line routines (Documentation/technical/protocol-common.txt)
82+
are used to simplify buffer management during message generation,
83+
transmission, and reception. A flush packet is used to mark the end
84+
of the message. This allows the sender to incrementally generate and
85+
transmit the message. It allows the receiver to incrementally receive
86+
the message in chunks and to know when they have received the entire
87+
message.
88+
89+
The actual byte format of the client request and server response
90+
messages are application specific. The IPC layer transmits and
91+
receives them as opaque byte buffers without any concern for the
92+
content within. It is the job of the calling application layer to
93+
understand the contents of the request and response messages.
94+
95+
96+
Summary
97+
-------
98+
99+
Conceptually, the Simple-IPC protocol is similar to an HTTP REST
100+
request. Clients connect, make an application-specific and
101+
stateless request, receive an application-specific
102+
response, and disconnect. It is a one round trip facility for
103+
querying the server. The Simple-IPC routines hide the socket,
104+
named pipe, and thread pool details and allow the application
105+
layer to focus on the application at hand.

‎Makefile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -467,6 +467,11 @@ all::
467467
# directory, and the JSON compilation database 'compile_commands.json' will be
468468
# created at the root of the repository.
469469
#
470+
# If your platform supports an built-in fsmonitor backend, set
471+
# FSMONITOR_DAEMON_BACKEND to the name of the corresponding
472+
# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the
473+
# `fsmonitor_fs_listen__*()` routines.
474+
#
470475
# Define DEVELOPER to enable more compiler warnings. Compiler version
471476
# and family are auto detected, but could be overridden by defining
472477
# COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -737,6 +742,7 @@ TEST_BUILTINS_OBJS += test-serve-v2.o
737742
TEST_BUILTINS_OBJS += test-sha1.o
738743
TEST_BUILTINS_OBJS += test-sha256.o
739744
TEST_BUILTINS_OBJS += test-sigchain.o
745+
TEST_BUILTINS_OBJS += test-simple-ipc.o
740746
TEST_BUILTINS_OBJS += test-strcmp-offset.o
741747
TEST_BUILTINS_OBJS += test-string-list.o
742748
TEST_BUILTINS_OBJS += test-submodule-config.o
@@ -883,6 +889,7 @@ LIB_OBJS += fetch-pack.o
883889
LIB_OBJS += fmt-merge-msg.o
884890
LIB_OBJS += fsck.o
885891
LIB_OBJS += fsmonitor.o
892+
LIB_OBJS += fsmonitor-ipc.o
886893
LIB_OBJS += gettext.o
887894
LIB_OBJS += gpg-interface.o
888895
LIB_OBJS += graph.o
@@ -1082,6 +1089,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o
10821089
BUILTIN_OBJS += builtin/for-each-ref.o
10831090
BUILTIN_OBJS += builtin/for-each-repo.o
10841091
BUILTIN_OBJS += builtin/fsck.o
1092+
BUILTIN_OBJS += builtin/fsmonitor--daemon.o
10851093
BUILTIN_OBJS += builtin/gc.o
10861094
BUILTIN_OBJS += builtin/get-tar-commit-id.o
10871095
BUILTIN_OBJS += builtin/grep.o
@@ -1672,6 +1680,14 @@ ifdef NO_UNIX_SOCKETS
16721680
BASIC_CFLAGS += -DNO_UNIX_SOCKETS
16731681
else
16741682
LIB_OBJS += unix-socket.o
1683+
LIB_OBJS += unix-stream-server.o
1684+
LIB_OBJS += compat/simple-ipc/ipc-shared.o
1685+
LIB_OBJS += compat/simple-ipc/ipc-unix-socket.o
1686+
endif
1687+
1688+
ifdef USE_WIN32_IPC
1689+
LIB_OBJS += compat/simple-ipc/ipc-shared.o
1690+
LIB_OBJS += compat/simple-ipc/ipc-win32.o
16751691
endif
16761692

16771693
ifdef NO_ICONV
@@ -1886,6 +1902,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER
18861902
COMPAT_OBJS += compat/access.o
18871903
endif
18881904

1905+
ifdef FSMONITOR_DAEMON_BACKEND
1906+
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
1907+
COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o
1908+
endif
1909+
18891910
ifeq ($(TCLTK_PATH),)
18901911
NO_TCLTK = NoThanks
18911912
endif
@@ -2736,6 +2757,9 @@ GIT-BUILD-OPTIONS: FORCE
27362757
@echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+
27372758
@echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
27382759
@echo X=\'$(X)\' >>$@+
2760+
ifdef FSMONITOR_DAEMON_BACKEND
2761+
@echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+
2762+
endif
27392763
ifdef TEST_OUTPUT_DIRECTORY
27402764
@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
27412765
endif

‎builtin.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);
158158
int cmd_for_each_repo(int argc, const char **argv, const char *prefix);
159159
int cmd_format_patch(int argc, const char **argv, const char *prefix);
160160
int cmd_fsck(int argc, const char **argv, const char *prefix);
161+
int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);
161162
int cmd_gc(int argc, const char **argv, const char *prefix);
162163
int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);
163164
int cmd_grep(int argc, const char **argv, const char *prefix);

‎builtin/credential-cache--daemon.c

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,9 +203,10 @@ static int serve_cache_loop(int fd)
203203

204204
static void serve_cache(const char *socket_path, int debug)
205205
{
206+
struct unix_stream_listen_opts opts = UNIX_STREAM_LISTEN_OPTS_INIT;
206207
int fd;
207208

208-
fd = unix_stream_listen(socket_path);
209+
fd = unix_stream_listen(socket_path, &opts);
209210
if (fd < 0)
210211
die_errno("unable to bind to '%s'", socket_path);
211212

0 commit comments

Comments
 (0)