Skip to content

Commit 47231c0

Browse files
dschodscho
committed
Merge branch 'fsmonitor4' of https://github.com/jeffhostetler/git
This branch thicket adds an built-in, Git-aware FSMonitor. Signed-off-by: Johannes Schindelin <[email protected]>
2 parents de685c9 + 1667f43 commit 47231c0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+7145
-75
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@
7171
/git-format-patch
7272
/git-fsck
7373
/git-fsck-objects
74+
/git-fsmonitor--daemon
7475
/git-gc
7576
/git-get-tar-commit-id
7677
/git-grep

‎Documentation/config/core.txt

Lines changed: 35 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -66,18 +66,43 @@ core.fsmonitor::
6666
will identify all files that may have changed since the
6767
requested date/time. This information is used to speed up git by
6868
avoiding unnecessary processing of files that have not changed.
69-
See the "fsmonitor-watchman" section of linkgit:githooks[5].
69+
+
70+
See the "fsmonitor-watchman" section of linkgit:githooks[5].
71+
+
72+
Note: FSMonitor hooks (and this config setting) are ignored if the
73+
built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).
7074

7175
core.fsmonitorHookVersion::
72-
Sets the version of hook that is to be used when calling fsmonitor.
73-
There are currently versions 1 and 2. When this is not set,
74-
version 2 will be tried first and if it fails then version 1
75-
will be tried. Version 1 uses a timestamp as input to determine
76-
which files have changes since that time but some monitors
77-
like watchman have race conditions when used with a timestamp.
78-
Version 2 uses an opaque string so that the monitor can return
79-
something that can be used to determine what files have changed
80-
without race conditions.
76+
Sets the version of hook that is to be used when calling the
77+
FSMonitor hook (as configured via `core.fsmonitor`).
78+
+
79+
There are currently versions 1 and 2. When this is not set,
80+
version 2 will be tried first and if it fails then version 1
81+
will be tried. Version 1 uses a timestamp as input to determine
82+
which files have changes since that time but some monitors
83+
like watchman have race conditions when used with a timestamp.
84+
Version 2 uses an opaque string so that the monitor can return
85+
something that can be used to determine what files have changed
86+
without race conditions.
87+
+
88+
Note: FSMonitor hooks (and this config setting) are ignored if the
89+
built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).
90+
91+
core.useBuiltinFSMonitor::
92+
If set to true, enable the built-in filesystem event watcher (for
93+
technical details, see linkgit:git-fsmonitor--daemon[1]).
94+
+
95+
Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up
96+
Git commands that need to refresh the Git index (e.g. `git status`) in a
97+
worktree with many files. The built-in FSMonitor facility eliminates the
98+
need to install and maintain an external third-party monitoring tool.
99+
+
100+
The built-in FSMonitor is currently available only on a limited set of
101+
supported platforms.
102+
+
103+
Note: if this config setting is set to `true`, any FSMonitor hook
104+
configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`)
105+
is ignored.
81106

82107
core.trustctime::
83108
If false, the ctime differences between the index and the

‎Documentation/git-fsmonitor--daemon.txt

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
git-fsmonitor--daemon(1)
2+
========================
3+
4+
NAME
5+
----
6+
git-fsmonitor--daemon - Builtin file system monitor daemon
7+
8+
SYNOPSIS
9+
--------
10+
[verse]
11+
'git fsmonitor--daemon' --start
12+
'git fsmonitor--daemon' --run
13+
'git fsmonitor--daemon' --stop
14+
'git fsmonitor--daemon' --is-running
15+
'git fsmonitor--daemon' --is-supported
16+
'git fsmonitor--daemon' --query <token>
17+
'git fsmonitor--daemon' --query-index
18+
'git fsmonitor--daemon' --flush
19+
20+
DESCRIPTION
21+
-----------
22+
23+
Monitors files and directories in the working directory for changes using
24+
platform-specific file system notification facilities.
25+
26+
It communicates directly with commands like `git status` using the
27+
link:technical/api-simple-ipc.html[simple IPC] interface instead of
28+
the slower linkgit:githooks[5] interface.
29+
30+
OPTIONS
31+
-------
32+
33+
--start::
34+
Starts the fsmonitor daemon in the background.
35+
36+
--run::
37+
Runs the fsmonitor daemon in the foreground.
38+
39+
--stop::
40+
Stops the fsmonitor daemon running for the current working
41+
directory, if present.
42+
43+
--is-running::
44+
Exits with zero status if the fsmonitor daemon is watching the
45+
current working directory.
46+
47+
--is-supported::
48+
Exits with zero status if the fsmonitor daemon feature is supported
49+
on this platform.
50+
51+
--query <token>::
52+
Connects to the fsmonitor daemon (starting it if necessary) and
53+
requests the list of changed files and directories since the
54+
given token.
55+
This is intended for testing purposes.
56+
57+
--query-index::
58+
Read the current `<token>` from the File System Monitor index
59+
extension (if present) and use it to query the fsmonitor daemon.
60+
This is intended for testing purposes.
61+
62+
--flush::
63+
Force the fsmonitor daemon to flush its in-memory cache and
64+
re-sync with the file system.
65+
This is intended for testing purposes.
66+
67+
REMARKS
68+
-------
69+
The fsmonitor daemon is a long running process that will watch a single
70+
working directory. Commands, such as `git status`, should automatically
71+
start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true`
72+
(see linkgit:git-config[1]).
73+
74+
Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each
75+
working directory separately, or globally via `git config --global
76+
core.useBuiltinFSMonitor true`.
77+
78+
Tokens are opaque strings. They are used by the fsmonitor daemon to
79+
mark a point in time and the associated internal state. Callers should
80+
make no assumptions about the content of the token. In particular,
81+
the should not assume that it is a timestamp.
82+
83+
Query commands send a request-token to the daemon and it responds with
84+
a summary of the changes that have occurred since that token was
85+
created. The daemon also returns a response-token that the client can
86+
use in a future query.
87+
88+
For more information see the "File System Monitor" section in
89+
linkgit:git-update-index[1].
90+
91+
CAVEATS
92+
-------
93+
94+
The fsmonitor daemon does not currently know about submodules and does
95+
not know to filter out file system events that happen within a
96+
submodule. If fsmonitor daemon is watching a super repo and a file is
97+
modified within the working directory of a submodule, it will report
98+
the change (as happening against the super repo). However, the client
99+
should properly ignore these extra events, so performance may be affected
100+
but it should not cause an incorrect result.
101+
102+
GIT
103+
---
104+
Part of the linkgit:git[1] suite

‎Documentation/git-update-index.txt

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR
498498
This feature is intended to speed up git operations for repos that have
499499
large working directories.
500500

501-
It enables git to work together with a file system monitor (see the
501+
It enables git to work together with a file system monitor (see
502+
linkgit:git-fsmonitor--daemon[1]
503+
and the
502504
"fsmonitor-watchman" section of linkgit:githooks[5]) that can
503505
inform it as to what files have been modified. This enables git to avoid
504506
having to lstat() every file to find modified files.

‎Documentation/githooks.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -584,7 +584,8 @@ fsmonitor-watchman
584584

585585
This hook is invoked when the configuration option `core.fsmonitor` is
586586
set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`
587-
depending on the version of the hook to use.
587+
depending on the version of the hook to use, unless overridden via
588+
`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).
588589

589590
Version 1 takes two arguments, a version (1) and the time in elapsed
590591
nanoseconds since midnight, January 1, 1970.

‎Documentation/technical/api-simple-ipc.txt

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
Simple-IPC API
2+
==============
3+
4+
The Simple-IPC API is a collection of `ipc_` prefixed library routines
5+
and a basic communication protocol that allow an IPC-client process to
6+
send an application-specific IPC-request message to an IPC-server
7+
process and receive an application-specific IPC-response message.
8+
9+
Communication occurs over a named pipe on Windows and a Unix domain
10+
socket on other platforms. IPC-clients and IPC-servers rendezvous at
11+
a previously agreed-to application-specific pathname (which is outside
12+
the scope of this design) that is local to the computer system.
13+
14+
The IPC-server routines within the server application process create a
15+
thread pool to listen for connections and receive request messages
16+
from multiple concurrent IPC-clients. When received, these messages
17+
are dispatched up to the server application callbacks for handling.
18+
IPC-server routines then incrementally relay responses back to the
19+
IPC-client.
20+
21+
The IPC-client routines within a client application process connect
22+
to the IPC-server and send a request message and wait for a response.
23+
When received, the response is returned back the caller.
24+
25+
For example, the `fsmonitor--daemon` feature will be built as a server
26+
application on top of the IPC-server library routines. It will have
27+
threads watching for file system events and a thread pool waiting for
28+
client connections. Clients, such as `git status` will request a list
29+
of file system events since a point in time and the server will
30+
respond with a list of changed files and directories. The formats of
31+
the request and response are application-specific; the IPC-client and
32+
IPC-server routines treat them as opaque byte streams.
33+
34+
35+
Comparison with sub-process model
36+
---------------------------------
37+
38+
The Simple-IPC mechanism differs from the existing `sub-process.c`
39+
model (Documentation/technical/long-running-process-protocol.txt) and
40+
used by applications like Git-LFS. In the LFS-style sub-process model
41+
the helper is started by the foreground process, communication happens
42+
via a pair of file descriptors bound to the stdin/stdout of the
43+
sub-process, the sub-process only serves the current foreground
44+
process, and the sub-process exits when the foreground process
45+
terminates.
46+
47+
In the Simple-IPC model the server is a very long-running service. It
48+
can service many clients at the same time and has a private socket or
49+
named pipe connection to each active client. It might be started
50+
(on-demand) by the current client process or it might have been
51+
started by a previous client or by the OS at boot time. The server
52+
process is not associated with a terminal and it persists after
53+
clients terminate. Clients do not have access to the stdin/stdout of
54+
the server process and therefore must communicate over sockets or
55+
named pipes.
56+
57+
58+
Server startup and shutdown
59+
---------------------------
60+
61+
How an application server based upon IPC-server is started is also
62+
outside the scope of the Simple-IPC design and is a property of the
63+
application using it. For example, the server might be started or
64+
restarted during routine maintenance operations, or it might be
65+
started as a system service during the system boot-up sequence, or it
66+
might be started on-demand by a foreground Git command when needed.
67+
68+
Similarly, server shutdown is a property of the application using
69+
the simple-ipc routines. For example, the server might decide to
70+
shutdown when idle or only upon explicit request.
71+
72+
73+
Simple-IPC protocol
74+
-------------------
75+
76+
The Simple-IPC protocol consists of a single request message from the
77+
client and an optional response message from the server. Both the
78+
client and server messages are unlimited in length and are terminated
79+
with a flush packet.
80+
81+
The pkt-line routines (Documentation/technical/protocol-common.txt)
82+
are used to simplify buffer management during message generation,
83+
transmission, and reception. A flush packet is used to mark the end
84+
of the message. This allows the sender to incrementally generate and
85+
transmit the message. It allows the receiver to incrementally receive
86+
the message in chunks and to know when they have received the entire
87+
message.
88+
89+
The actual byte format of the client request and server response
90+
messages are application specific. The IPC layer transmits and
91+
receives them as opaque byte buffers without any concern for the
92+
content within. It is the job of the calling application layer to
93+
understand the contents of the request and response messages.
94+
95+
96+
Summary
97+
-------
98+
99+
Conceptually, the Simple-IPC protocol is similar to an HTTP REST
100+
request. Clients connect, make an application-specific and
101+
stateless request, receive an application-specific
102+
response, and disconnect. It is a one round trip facility for
103+
querying the server. The Simple-IPC routines hide the socket,
104+
named pipe, and thread pool details and allow the application
105+
layer to focus on the application at hand.

‎Makefile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -467,6 +467,11 @@ all::
467467
# directory, and the JSON compilation database 'compile_commands.json' will be
468468
# created at the root of the repository.
469469
#
470+
# If your platform supports an built-in fsmonitor backend, set
471+
# FSMONITOR_DAEMON_BACKEND to the name of the corresponding
472+
# `compat/fsmonitor/fsmonitor-fs-listen-<name>.c` that implements the
473+
# `fsmonitor_fs_listen__*()` routines.
474+
#
470475
# Define DEVELOPER to enable more compiler warnings. Compiler version
471476
# and family are auto detected, but could be overridden by defining
472477
# COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -737,6 +742,7 @@ TEST_BUILTINS_OBJS += test-serve-v2.o
737742
TEST_BUILTINS_OBJS += test-sha1.o
738743
TEST_BUILTINS_OBJS += test-sha256.o
739744
TEST_BUILTINS_OBJS += test-sigchain.o
745+
TEST_BUILTINS_OBJS += test-simple-ipc.o
740746
TEST_BUILTINS_OBJS += test-strcmp-offset.o
741747
TEST_BUILTINS_OBJS += test-string-list.o
742748
TEST_BUILTINS_OBJS += test-submodule-config.o
@@ -883,6 +889,7 @@ LIB_OBJS += fetch-pack.o
883889
LIB_OBJS += fmt-merge-msg.o
884890
LIB_OBJS += fsck.o
885891
LIB_OBJS += fsmonitor.o
892+
LIB_OBJS += fsmonitor-ipc.o
886893
LIB_OBJS += gettext.o
887894
LIB_OBJS += gpg-interface.o
888895
LIB_OBJS += graph.o
@@ -1082,6 +1089,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o
10821089
BUILTIN_OBJS += builtin/for-each-ref.o
10831090
BUILTIN_OBJS += builtin/for-each-repo.o
10841091
BUILTIN_OBJS += builtin/fsck.o
1092+
BUILTIN_OBJS += builtin/fsmonitor--daemon.o
10851093
BUILTIN_OBJS += builtin/gc.o
10861094
BUILTIN_OBJS += builtin/get-tar-commit-id.o
10871095
BUILTIN_OBJS += builtin/grep.o
@@ -1672,6 +1680,14 @@ ifdef NO_UNIX_SOCKETS
16721680
BASIC_CFLAGS += -DNO_UNIX_SOCKETS
16731681
else
16741682
LIB_OBJS += unix-socket.o
1683+
LIB_OBJS += unix-stream-server.o
1684+
LIB_OBJS += compat/simple-ipc/ipc-shared.o
1685+
LIB_OBJS += compat/simple-ipc/ipc-unix-socket.o
1686+
endif
1687+
1688+
ifdef USE_WIN32_IPC
1689+
LIB_OBJS += compat/simple-ipc/ipc-shared.o
1690+
LIB_OBJS += compat/simple-ipc/ipc-win32.o
16751691
endif
16761692

16771693
ifdef NO_ICONV
@@ -1886,6 +1902,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER
18861902
COMPAT_OBJS += compat/access.o
18871903
endif
18881904

1905+
ifdef FSMONITOR_DAEMON_BACKEND
1906+
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
1907+
COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o
1908+
endif
1909+
18891910
ifeq ($(TCLTK_PATH),)
18901911
NO_TCLTK = NoThanks
18911912
endif
@@ -2729,6 +2750,9 @@ GIT-BUILD-OPTIONS: FORCE
27292750
@echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+
27302751
@echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
27312752
@echo X=\'$(X)\' >>$@+
2753+
ifdef FSMONITOR_DAEMON_BACKEND
2754+
@echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+
2755+
endif
27322756
ifdef TEST_OUTPUT_DIRECTORY
27332757
@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
27342758
endif

‎builtin.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);
158158
int cmd_for_each_repo(int argc, const char **argv, const char *prefix);
159159
int cmd_format_patch(int argc, const char **argv, const char *prefix);
160160
int cmd_fsck(int argc, const char **argv, const char *prefix);
161+
int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);
161162
int cmd_gc(int argc, const char **argv, const char *prefix);
162163
int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);
163164
int cmd_grep(int argc, const char **argv, const char *prefix);

‎builtin/credential-cache--daemon.c

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,9 +203,10 @@ static int serve_cache_loop(int fd)
203203

204204
static void serve_cache(const char *socket_path, int debug)
205205
{
206+
struct unix_stream_listen_opts opts = UNIX_STREAM_LISTEN_OPTS_INIT;
206207
int fd;
207208

208-
fd = unix_stream_listen(socket_path);
209+
fd = unix_stream_listen(socket_path, &opts);
209210
if (fd < 0)
210211
die_errno("unable to bind to '%s'", socket_path);
211212

‎builtin/credential-cache.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
static int send_request(const char *socket, const struct strbuf *out)
1515
{
1616
int got_data = 0;
17-
int fd = unix_stream_connect(socket);
17+
int fd = unix_stream_connect(socket, 0);
1818

1919
if (fd < 0)
2020
return -1;

0 commit comments

Comments
 (0)