Skip to content

bpo-30860: Consolidate stateful runtime globals. #3397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 94 commits into from
Sep 8, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
d4799a7
Add the check-c-globals script.
ericsnowcurrently May 31, 2017
9abaa7b
Clean up the check-c-globals script.
ericsnowcurrently Jun 1, 2017
aa3a7db
Update the list of "runtime" globals.
ericsnowcurrently Jun 1, 2017
c7c86c7
Handle builtin modules via a pattern check.
ericsnowcurrently Jun 1, 2017
6b991d9
Rely on a pattern check for static type-related vars.
ericsnowcurrently Jun 1, 2017
8a8763d
Handle most exceptions and warnings via pattern checks.
ericsnowcurrently Jun 1, 2017
c8b5b43
Clean up _is_type_var().
ericsnowcurrently Jun 1, 2017
72e1d9b
Add runtime vars for builtin modules.
ericsnowcurrently Jun 1, 2017
f51e788
Ignore trailing comments in globals-runtime.txt.
ericsnowcurrently Jun 1, 2017
9836224
Add runtime vars for builtin types.
ericsnowcurrently Jun 1, 2017
9dab7a7
Add runtime vars for compiler.
ericsnowcurrently Jun 1, 2017
7e1cad8
Add runtime vars for interpreter.
ericsnowcurrently Jun 1, 2017
7b8279a
Add runtime vars for import.
ericsnowcurrently Jun 1, 2017
c067341
Add runtime vars for startup.
ericsnowcurrently Jun 1, 2017
794fb24
Add other runtime vars.
ericsnowcurrently Jun 1, 2017
942dbfa
Separate the 3 runtime global "scopes".
ericsnowcurrently Jun 1, 2017
7a2c42a
Do not worry about a possible race.
ericsnowcurrently Jun 1, 2017
2e6526b
Narrow down the globals lists.
ericsnowcurrently Jun 3, 2017
39296ea
runtime -> core.
ericsnowcurrently Jun 3, 2017
103205d
Hold up more globals.
ericsnowcurrently Jun 7, 2017
ea3a2cf
Add _PyRuntimeState and _PyRuntime.
ericsnowcurrently Jun 19, 2017
a112d71
Pull pylifecycle globals into _PyRuntimeState.
ericsnowcurrently Jun 19, 2017
99a8aa6
Pull mem globals into _PyRuntimeState.
ericsnowcurrently Jun 20, 2017
4e6d1d9
Pull warnings globals into _PyRuntimeState.
ericsnowcurrently Jun 21, 2017
10f7601
Move nb_threads into PyInterpreterState.
ericsnowcurrently Jun 21, 2017
d82561c
Pull ceval globals into _PyRuntimeState.
ericsnowcurrently Jun 21, 2017
06b8fc8
Pull gil into _PyRuntimeState.
ericsnowcurrently Jun 21, 2017
960e1b0
Move check_interval, warnoptions, and xoptions into PyInterpreterState.
ericsnowcurrently Jun 21, 2017
8fb060f
Move pythread_stacksize into PyInterpreterState.
ericsnowcurrently Jun 21, 2017
fc8d4fd
Pull the rest of the GIL globals into _PyRuntime.
ericsnowcurrently Jun 28, 2017
1189e2e
Factor out _gil_globals and _gilstate_globals structs.
ericsnowcurrently Jun 28, 2017
edbaad5
Factor out globals structs.
ericsnowcurrently Jun 28, 2017
cec09bf
Factor out a _pending_calls struct.
ericsnowcurrently Jun 29, 2017
4dc9692
Update globals list.
ericsnowcurrently Jun 29, 2017
c438b8f
Ignore a few more globals.
ericsnowcurrently Jun 30, 2017
d0dd810
Only track one list of globals.
ericsnowcurrently Jul 3, 2017
d8e5c21
Mark _Py_HashSecret* as global.
ericsnowcurrently Jul 3, 2017
1e48db2
_Py_TracingPossible -> _PyRuntime.ceval.tracing_possible.
ericsnowcurrently Jul 3, 2017
6a1f65f
Re-structure globals.txt.
ericsnowcurrently Jul 3, 2017
f41754f
Ignore the symtable globals.
ericsnowcurrently Jul 3, 2017
cb29c94
_Py_CoreInitialized -> _PyRuntime.core_initialized.
ericsnowcurrently Jul 3, 2017
e1f1c4c
Pull in the list of interpreters.
ericsnowcurrently Jul 3, 2017
4befb69
Clean up globals.txt.
ericsnowcurrently Jul 3, 2017
ed94b6a
Ignore the import lock.
ericsnowcurrently Jul 3, 2017
cbccbf2
Fix func header formatting.
ericsnowcurrently Jul 4, 2017
41ce494
Move "globals" structs to the appropriate header files.
ericsnowcurrently Jul 5, 2017
6ef10c9
Factor out pymalloc.h.
ericsnowcurrently Jul 4, 2017
a8263c3
Pull usedpools into _PyRuntime.
ericsnowcurrently Jul 5, 2017
b4482c7
Drop a dead comment.
ericsnowcurrently Jul 5, 2017
ed25669
Pull in gcmodule.c.
ericsnowcurrently Jul 5, 2017
f6e4b9c
Clean up globals.txt.
ericsnowcurrently Jul 5, 2017
0817b0e
Pass pointers to the *_Initialize() funcs.
ericsnowcurrently Jul 5, 2017
4eea51e
Clean up line length.
ericsnowcurrently Jul 6, 2017
40292bd
Add a filename CLI arg.
ericsnowcurrently Jul 6, 2017
16b0b73
Move check-c-globals into Tools/c-globals.
ericsnowcurrently Jul 6, 2017
1b137d1
Add a README.
ericsnowcurrently Jul 6, 2017
5cbb8ed
Fix typo in header guard.
ericsnowcurrently Jul 6, 2017
46e4063
Ignore _TARGET_LOCALES.
ericsnowcurrently Jul 6, 2017
1074895
Factor out _PyRuntimeState_Initialize().
ericsnowcurrently Jul 7, 2017
9dbb1d1
Only initialize _PyRuntime once.
ericsnowcurrently Jul 7, 2017
c1d73b4
Do not assume that there is an active threadstate.
ericsnowcurrently Jul 8, 2017
18b5b18
Always use the global allocators.
ericsnowcurrently Jul 8, 2017
29d21bd
Factor out _allocators_globals.
ericsnowcurrently Jul 8, 2017
3eb1fa9
Touch up the c-globals README.
ericsnowcurrently Jul 9, 2017
3063db1
globals.txt -> ignored-globals.txt.
ericsnowcurrently Jul 9, 2017
bc3b84b
Move runtime state into "internal" header files.
ericsnowcurrently Jul 10, 2017
e411f86
globals -> state
ericsnowcurrently Jul 11, 2017
330552f
Define _PyRuntime for pgen.
ericsnowcurrently Jul 11, 2017
a18d6f5
Drop a superfluous include.
ericsnowcurrently Jul 11, 2017
2434213
Add the new header files to the Windows config.
ericsnowcurrently Jul 11, 2017
8e9deb7
Move _PyInterpreterState_Init() to the internal header file.
ericsnowcurrently Jul 12, 2017
671eeef
Add _PyRuntimeState_Fini().
ericsnowcurrently Jul 12, 2017
aed8c9a
Manage the interpreters mutex globally.
ericsnowcurrently Jul 12, 2017
bdb5ced
Fix Windows build.
ericsnowcurrently Jul 14, 2017
aaab3a9
"uint" -> unsigned int.
ericsnowcurrently Jul 17, 2017
1903528
Set up allocators in _Py_InitializeCore().
ericsnowcurrently Jul 17, 2017
be04984
Add a NEWS entry.
ericsnowcurrently Sep 5, 2017
5d3b366
Fix the news entry.
ericsnowcurrently Sep 5, 2017
8e6fcc0
Drop the underscore prefix from include file names.
ericsnowcurrently Sep 6, 2017
91d239c
Move _PyRuntime into its own header file.
ericsnowcurrently Sep 6, 2017
b223e85
Move _PyRuntime back to internal/pystate.h.
ericsnowcurrently Sep 6, 2017
3d0b18b
Include internal/pystate.h directly in source files.
ericsnowcurrently Sep 6, 2017
66c0114
Fix pgen.
ericsnowcurrently Sep 6, 2017
bb27081
Fix empty initializer lists.
ericsnowcurrently Sep 6, 2017
4797ce5
Fix includes.
ericsnowcurrently Sep 6, 2017
6592065
Drop an insignificant comment.
ericsnowcurrently Sep 7, 2017
69fe645
Restore _Py_CheckRecursionLimit under the stable ABI.
ericsnowcurrently Sep 7, 2017
6628e85
_Py_IS_FINALIZING() -> _Py_IsFinalizing().
ericsnowcurrently Sep 7, 2017
0f97249
Fix modules on Windows.
ericsnowcurrently Sep 7, 2017
1c85e28
Drop WITH_THREAD.
ericsnowcurrently Sep 7, 2017
2b28324
Undef Py_BUILD_CORE in core extension modules.
ericsnowcurrently Sep 7, 2017
b1ca420
Fix a syntax error in a Windows-only file.
ericsnowcurrently Sep 7, 2017
72d7251
Be careful about Yield on Windows.
ericsnowcurrently Sep 7, 2017
f1f10ff
Fix includes in builtin extensions on Windows.
ericsnowcurrently Sep 8, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Include/ceval.h
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ PyAPI_FUNC(int) Py_GetRecursionLimit(void);
PyThreadState_GET()->overflowed = 0; \
} while(0)
PyAPI_FUNC(int) _Py_CheckRecursiveCall(const char *where);
/* XXX _Py_CheckRecursionLimit should be changed to
_PyRuntime.ceval.check_recursion_limit. However, due to the macros
in which it's used, _Py_CheckRecursionLimit is stuck in the stable
ABI. It should be removed therefrom when possible.
*/
PyAPI_DATA(int) _Py_CheckRecursionLimit;

#ifdef USE_STACKCHECK
Expand Down
53 changes: 53 additions & 0 deletions Include/internal/ceval.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#ifndef Py_INTERNAL_CEVAL_H
#define Py_INTERNAL_CEVAL_H
#ifdef __cplusplus
extern "C" {
#endif

#include "pyatomic.h"
#include "pythread.h"

struct _pending_calls {
unsigned long main_thread;
PyThread_type_lock lock;
/* Request for running pending calls. */
_Py_atomic_int calls_to_do;
/* Request for looking at the `async_exc` field of the current
thread state.
Guarded by the GIL. */
int async_exc;
#define NPENDINGCALLS 32
struct {
int (*func)(void *);
void *arg;
} calls[NPENDINGCALLS];
int first;
int last;
};

#include "internal/gil.h"

struct _ceval_runtime_state {
int recursion_limit;
int check_recursion_limit;
/* Records whether tracing is on for any thread. Counts the number
of threads for which tstate->c_tracefunc is non-NULL, so if the
value is 0, we know we don't have to check this thread's
c_tracefunc. This speeds up the if statement in
PyEval_EvalFrameEx() after fast_next_opcode. */
int tracing_possible;
/* This single variable consolidates all requests to break out of
the fast path in the eval loop. */
_Py_atomic_int eval_breaker;
/* Request for dropping the GIL */
_Py_atomic_int gil_drop_request;
struct _pending_calls pending;
struct _gil_runtime_state gil;
};

PyAPI_FUNC(void) _PyEval_Initialize(struct _ceval_runtime_state *);

#ifdef __cplusplus
}
#endif
#endif /* !Py_INTERNAL_CEVAL_H */
91 changes: 91 additions & 0 deletions Include/internal/condvar.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#ifndef Py_INTERNAL_CONDVAR_H
#define Py_INTERNAL_CONDVAR_H

#ifndef _POSIX_THREADS
/* This means pthreads are not implemented in libc headers, hence the macro
not present in unistd.h. But they still can be implemented as an external
library (e.g. gnu pth in pthread emulation) */
# ifdef HAVE_PTHREAD_H
# include <pthread.h> /* _POSIX_THREADS */
# endif
#endif

#ifdef _POSIX_THREADS
/*
* POSIX support
*/
#define Py_HAVE_CONDVAR

#include <pthread.h>

#define PyMUTEX_T pthread_mutex_t
#define PyCOND_T pthread_cond_t

#elif defined(NT_THREADS)
/*
* Windows (XP, 2003 server and later, as well as (hopefully) CE) support
*
* Emulated condition variables ones that work with XP and later, plus
* example native support on VISTA and onwards.
*/
#define Py_HAVE_CONDVAR

/* include windows if it hasn't been done before */
#define WIN32_LEAN_AND_MEAN
#include <windows.h>

/* options */
/* non-emulated condition variables are provided for those that want
* to target Windows Vista. Modify this macro to enable them.
*/
#ifndef _PY_EMULATED_WIN_CV
#define _PY_EMULATED_WIN_CV 1 /* use emulated condition variables */
#endif

/* fall back to emulation if not targeting Vista */
#if !defined NTDDI_VISTA || NTDDI_VERSION < NTDDI_VISTA
#undef _PY_EMULATED_WIN_CV
#define _PY_EMULATED_WIN_CV 1
#endif

#if _PY_EMULATED_WIN_CV

typedef CRITICAL_SECTION PyMUTEX_T;

/* The ConditionVariable object. From XP onwards it is easily emulated
with a Semaphore.
Semaphores are available on Windows XP (2003 server) and later.
We use a Semaphore rather than an auto-reset event, because although
an auto-resent event might appear to solve the lost-wakeup bug (race
condition between releasing the outer lock and waiting) because it
maintains state even though a wait hasn't happened, there is still
a lost wakeup problem if more than one thread are interrupted in the
critical place. A semaphore solves that, because its state is
counted, not Boolean.
Because it is ok to signal a condition variable with no one
waiting, we need to keep track of the number of
waiting threads. Otherwise, the semaphore's state could rise
without bound. This also helps reduce the number of "spurious wakeups"
that would otherwise happen.
*/

typedef struct _PyCOND_T
{
HANDLE sem;
int waiting; /* to allow PyCOND_SIGNAL to be a no-op */
} PyCOND_T;

#else /* !_PY_EMULATED_WIN_CV */

/* Use native Win7 primitives if build target is Win7 or higher */

/* SRWLOCK is faster and better than CriticalSection */
typedef SRWLOCK PyMUTEX_T;

typedef CONDITION_VARIABLE PyCOND_T;

#endif /* _PY_EMULATED_WIN_CV */

#endif /* _POSIX_THREADS, NT_THREADS */

#endif /* Py_INTERNAL_CONDVAR_H */
46 changes: 46 additions & 0 deletions Include/internal/gil.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#ifndef Py_INTERNAL_GIL_H
#define Py_INTERNAL_GIL_H
#ifdef __cplusplus
extern "C" {
#endif

#include "pyatomic.h"

#include "internal/condvar.h"
#ifndef Py_HAVE_CONDVAR
#error You need either a POSIX-compatible or a Windows system!
#endif

/* Enable if you want to force the switching of threads at least
every `interval`. */
#undef FORCE_SWITCHING
#define FORCE_SWITCHING

struct _gil_runtime_state {
/* microseconds (the Python API uses seconds, though) */
unsigned long interval;
/* Last PyThreadState holding / having held the GIL. This helps us
know whether anyone else was scheduled after we dropped the GIL. */
_Py_atomic_address last_holder;
/* Whether the GIL is already taken (-1 if uninitialized). This is
atomic because it can be read without any lock taken in ceval.c. */
_Py_atomic_int locked;
/* Number of GIL switches since the beginning. */
unsigned long switch_number;
/* This condition variable allows one or several threads to wait
until the GIL is released. In addition, the mutex also protects
the above variables. */
PyCOND_T cond;
PyMUTEX_T mutex;
#ifdef FORCE_SWITCHING
/* This condition variable helps the GIL-releasing thread wait for
a GIL-awaiting thread to be scheduled and take the GIL. */
PyCOND_T switch_cond;
PyMUTEX_T switch_mutex;
#endif
};

#ifdef __cplusplus
}
#endif
#endif /* !Py_INTERNAL_GIL_H */
Loading