[Executorch][threadpool] Add "private" method to reset num threads in threadpool

Pull Request resolved: #2351

This diff introduces new API to reset threadpool. This will be short-lived
features. It is needed atm because we dont have a) a way to use only a subset of
threads to distribute work in the threadpool and b) API to create threadpool of
custom size and pass such a threadpool to backends like delegtes, cusotm ops,
optimized cpu op lib etc.
ghstack-source-id: 218819088
@exported-using-ghexport

Differential Revision: [D54766070](https://our.internmc.facebook.com/intern/diff/D54766070/)

Author: kimishpatel

backends/xnnpack/threadpool/threadpool.cpp

@@ -44,6 +44,18 @@ size_t ThreadPool::get_thread_count() const {
   return pthreadpool_get_threads_count(threadpool_.get());
 }
 
+bool ThreadPool::_unsafe_reset_threadpool(uint32_t new_thread_count) {
+  // No need to do anything if the count is same
+  if (new_thread_count == get_thread_count()) {
+    return true;
+  }
+
+  std::lock_guard<std::mutex> lock{mutex_};
+
+  threadpool_.reset(pthreadpool_create(new_thread_count));
+  return true;
+}
+
 void ThreadPool::run(
     const std::function<void(size_t)>& fn,
     const size_t range) {

backends/xnnpack/threadpool/threadpool.h

@@ -34,6 +34,17 @@ class ThreadPool final {
 
   size_t get_thread_count() const;
 
+  /*
+   * Resets the threadpool by creating a new threadpool with requested # of
+   * threads. This is not a thread safe call. When calling this method, threads
+   * of the threadpool might be doing some work. Some other code may also be
+   * holding on to the threadpool pointer, that is no longer valid. This is a
+   * private API, which will later be replaced by something that allows creating
+   * of threadpool with requested size and use such a threadpool with backend
+   * delegates, custom ops or optimized lib.
+   */
+  bool _unsafe_reset_threadpool(uint32_t num_threads);
+
   // Run, in parallel, function fn(task_id) over task_id in range [0, range).
   // This function is blocking.  All input is processed by the time it returns.
   // NoThreadPoolGuard (see threadpool_guard.h) can used to disable