Port fixes for second point release. (#10146)

* Better messaging on notebook fail (#10056)

* Install jupyter instead of installing kernel spec (#10080)

* Install jupyter instead of installing kernel spec
For #10071

* Eliminate variable value when computing data frame info (#10081)

* Fix ndarray types to be viewable again (#10093)

* Eliminate variable value when computing data frame info

* Fix ndarrays to work again
Add test to make sure we don't regress this again

* Rchiodo/kernel telemetry (#10115)

* Add duration to select local/remote kernel

* Add notebook language telemetry

* Add news entries

* #9883 telemetry

* News entry

* Another spot for kernel spec failure

* Add telemetry on product install

* Fix install telemetry

* Undo launch.json change

* Handle other cases

* Better way to handle case

* Wrong event for jupyter install

* Fix unit tests

* Clear variables when restarting regardless if visible or not (#10117)

* Use different method for checking if kernelspec is available (#10114)

* Use different method for checking if kernelspec is available

* Fix unit tests

* More logging for kernelspec problems (#10132)

* More logging for kernelspec problems

* Actually capture the exception on the new code

* Not actually using output if first exception still there.

* Actually only return output on one of the expected calls.

* Fix nightly flake

* Check our saved jupyter interpreters before allowing any of them to be used as active interpreters (#10113)

* Update changelog and package.json

* Missing part of changelog

* Fix tests and linter problems

Co-authored-by: Ian Huff <[email protected]>
Co-authored-by: Don Jayamanne <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -1,5 +1,82 @@
 # Changelog
 
+## 2020.2.2 (18 February 2020)
+
+### Fixes
+
+1. Improve error messaging when the jupyter notebook cannot be started.
+   ([#9904](https://github.com/Microsoft/vscode-python/issues/9904))
+1. Clear variables in notebooks and interactive-window when restarting.
+   ([#9991](https://github.com/Microsoft/vscode-python/issues/9991))
+1. Re-install `Jupyter` instead of installing `kernelspec` if `kernelspec` cannot be found in the python environment.
+   ([#10071](https://github.com/Microsoft/vscode-python/issues/10071))
+1. Fixes problem with showing ndarrays in the data viewer.
+   ([#10074](https://github.com/Microsoft/vscode-python/issues/10074))
+1. Fix data viewer not opening on certain data frames.
+   ([#10075](https://github.com/Microsoft/vscode-python/issues/10075))
+
+### Code Health
+
+1. Add telemetry to track notebook languages
+   ([#9819](https://github.com/Microsoft/vscode-python/issues/9819))
+1. Telemetry around kernels not working and installs not working.
+   ([#9883](https://github.com/Microsoft/vscode-python/issues/9883))
+1. Change select kernel telemetry to track duration till quick pick appears.
+   ([#10049](https://github.com/Microsoft/vscode-python/issues/10049))
+
+### Thanks
+
+Thanks to the following projects which we fully rely on to provide some of
+our features:
+
+-   [isort](https://pypi.org/project/isort/)
+-   [jedi](https://pypi.org/project/jedi/)
+    and [parso](https://pypi.org/project/parso/)
+-   [Microsoft Python Language Server](https://github.com/microsoft/python-language-server)
+-   [ptvsd](https://pypi.org/project/ptvsd/)
+-   [exuberant ctags](http://ctags.sourceforge.net/) (user-installed)
+-   [rope](https://pypi.org/project/rope/) (user-installed)
+
+Also thanks to the various projects we provide integrations with which help
+make this extension useful:
+
+-   Debugging support:
+    [Django](https://pypi.org/project/Django/),
+    [Flask](https://pypi.org/project/Flask/),
+    [gevent](https://pypi.org/project/gevent/),
+    [Jinja](https://pypi.org/project/Jinja/),
+    [Pyramid](https://pypi.org/project/pyramid/),
+    [PySpark](https://pypi.org/project/pyspark/),
+    [Scrapy](https://pypi.org/project/Scrapy/),
+    [Watson](https://pypi.org/project/Watson/)
+-   Formatting:
+    [autopep8](https://pypi.org/project/autopep8/),
+    [black](https://pypi.org/project/black/),
+    [yapf](https://pypi.org/project/yapf/)
+-   Interpreter support:
+    [conda](https://conda.io/),
+    [direnv](https://direnv.net/),
+    [pipenv](https://pypi.org/project/pipenv/),
+    [pyenv](https://github.com/pyenv/pyenv),
+    [venv](https://docs.python.org/3/library/venv.html#module-venv),
+    [virtualenv](https://pypi.org/project/virtualenv/)
+-   Linting:
+    [bandit](https://pypi.org/project/bandit/),
+    [flake8](https://pypi.org/project/flake8/),
+    [mypy](https://pypi.org/project/mypy/),
+    [prospector](https://pypi.org/project/prospector/),
+    [pylint](https://pypi.org/project/pylint/),
+    [pydocstyle](https://pypi.org/project/pydocstyle/),
+    [pylama](https://pypi.org/project/pylama/)
+-   Testing:
+    [nose](https://pypi.org/project/nose/),
+    [pytest](https://pypi.org/project/pytest/),
+    [unittest](https://docs.python.org/3/library/unittest.html#module-unittest)
+
+And finally thanks to the [Python](https://www.python.org/) development team and
+community for creating a fantastic programming language and community to be a
+part of!
+
 ## 2020.2.1 (12 February 2020)
 
 ### Fixes

PYTHON_INTERACTIVE_TROUBLESHOOTING.md

@@ -1,6 +1,6 @@
-# Trouble shooting the Python Interactive Window
+# Troubleshooting Jupyter issues in the Python Interactive Window or Notebook Editor
 
-This document is intended to help troubleshoot problems in the Python Interactive Window.
+This document is intended to help troubleshoot problems with starting Jupyter in the Python Interactive Window or Notebook Editor.
 
 ---
 ## Jupyter Not Starting
@@ -10,12 +10,21 @@ This error can happen when
 * Jupyter is not installed
 * You picked the wrong Python environment (one that doesn't have Jupyter installed).
 
-### The first step is to verify you are running the Python environment you want.
+### The first step is to verify you are running the Python environment that you have Jupyter installed into.
 
-The Python you're using is picked with the selection dropdown on the bottom left of the VS Code window:
+The first time that you start the Interactive Window or the Notebook Editor VS Code will attempt to locate a Python environment that has Jupyter installed in it and can start a notebook. 
+
+The first Python interpreter to check will be the one selected with the selection dropdown on the bottom left of the VS Code window:
 
 ![selector](resources/PythonSelector.png)
 
+Once a suitable interpreter with Jupyter has been located, VS Code will continue to use that interpreter for starting up Jupyter servers.
+If no interpreters are found with Jupyter installed a popup message will ask if you would like to install Jupyter into the current interpreter.
+
+![install Jupyter](resources/InstallJupyter.png)
+
+If you would like to change from using the saved Python interpreter to a new interpreter for launching Jupyter just use the "Python: Select interpreter to start Jupyter server" VS Code command to change it.
+
 ### The second step is to check that jupyter isn't giving any errors on startup.
 
 Run the following command from an environment that matches the Python you selected:

package.json

@@ -2,7 +2,7 @@
     "name": "python",
     "displayName": "Python",
     "description": "Linting, Debugging (multi-threaded, remote), Intellisense, Jupyter Notebooks, code formatting, refactoring, unit tests, snippets, and more.",
-    "version": "2020.2.1",
+    "version": "2020.2.2",
     "languageServerVersion": "0.5.30",
     "publisher": "ms-python",
     "author": {

package.nls.json

@@ -193,6 +193,8 @@
     "DataScience.libraryNotInstalled": "Data Science library {0} is not installed. Install?",
     "DataScience.libraryRequiredToLaunchJupyterNotInstalled": "Data Science library {0} is not installed.",
     "DataScience.librariesRequiredToLaunchJupyterNotInstalled": "Data Science libraries {0} are not installed.",
+    "DataScience.libraryRequiredToLaunchJupyterNotInstalledInterpreter": "Data Science library {1} is not installed in interpreter {0}.",
+    "DataScience.librariesRequiredToLaunchJupyterNotInstalledInterpreter": "Data Science libraries {1} are not installed in interpreter {0}.",
     "DataScience.jupyterInstall": "Install",
     "DataScience.jupyterSelectURIPrompt": "Enter the URI of the running Jupyter server",
     "DataScience.jupyterSelectURIInvalidURI": "Invalid URI specified",

pythonFiles/datascience/getJupyterKernelspecVersion.py

@@ -0,0 +1,7 @@
+# Check whether kernelspec module exists.
+import sys
+import jupyter_client
+import jupyter_client.kernelspec
+
+sys.stdout.write(jupyter_client.__version__)
+sys.stdout.flush()

resources/InstallJupyter.png

@@ -5,6 +5,7 @@ import * as os from 'os';
 import { CancellationToken, OutputChannel, Uri } from 'vscode';
 import '../../common/extensions';
 import * as localize from '../../common/utils/localize';
+import { Telemetry } from '../../datascience/constants';
 import { IInterpreterService } from '../../interpreter/contracts';
 import { IServiceContainer } from '../../ioc/types';
 import { LinterId } from '../../linters/types';
@@ -18,6 +19,7 @@ import { IProcessServiceFactory, IPythonExecutionFactory } from '../process/type
 import { ITerminalServiceFactory } from '../terminal/types';
 import { IConfigurationService, IInstaller, InstallerResponse, IOutputChannel, IPersistentStateFactory, ModuleNamePurpose, Product, ProductType } from '../types';
 import { isResource } from '../utils/misc';
+import { StopWatch } from '../utils/stopWatch';
 import { ProductNames } from './productNames';
 import { IInstallationChannelManager, InterpreterUri, IProductPathService, IProductService } from './types';
 
@@ -278,7 +280,21 @@ export class DataScienceInstaller extends BaseInstaller {
     protected async promptToInstallImplementation(product: Product, resource?: InterpreterUri, cancel?: CancellationToken): Promise<InstallerResponse> {
         const productName = ProductNames.get(product)!;
         const item = await this.appShell.showErrorMessage(localize.DataScience.libraryNotInstalled().format(productName), 'Yes', 'No');
-        return item === 'Yes' ? this.install(product, resource, cancel) : InstallerResponse.Ignore;
+        if (item === 'Yes') {
+            const stopWatch = new StopWatch();
+            try {
+                const response = await this.install(product, resource, cancel);
+                const event = product === Product.jupyter ? Telemetry.UserInstalledJupyter : Telemetry.UserInstalledModule;
+                sendTelemetryEvent(event, stopWatch.elapsedTime, { product: productName });
+                return response;
+            } catch (e) {
+                if (product === Product.jupyter) {
+                    sendTelemetryEvent(Telemetry.JupyterInstallFailed);
+                }
+                throw e;
+            }
+        }
+        return InstallerResponse.Ignore;
     }
 }
 

src/client/common/installer/productInstaller.ts

@@ -113,8 +113,8 @@ export interface IPythonExecutionFactory {
     create(options: ExecutionFactoryCreationOptions): Promise<IPythonExecutionService>;
     /**
      * Creates a daemon Python Process.
-     * On windows its cheapter to create a daemon and use that than spin up Python Processes everytime.
-     * If something cannot be executed within the daemin, it will resort to using the stanard IPythonExecutionService.
+     * On windows it's cheaper to create a daemon and use that than spin up Python Processes everytime.
+     * If something cannot be executed within the daemon, it will resort to using the standard IPythonExecutionService.
      * Note: The returned execution service is always using an activated environment.
      *
      * @param {ExecutionFactoryCreationOptions} options

src/client/common/process/types.ts

@@ -211,6 +211,14 @@ export namespace DataScience {
         'DataScience.librariesRequiredToLaunchJupyterNotInstalled',
         'Data Science libraries {0} are not installed.'
     );
+    export const libraryRequiredToLaunchJupyterNotInstalledInterpreter = localize(
+        'DataScience.libraryRequiredToLaunchJupyterNotInstalledInterpreter',
+        'Data Science library {1} is not installed in interpreter {0}.'
+    );
+    export const librariesRequiredToLaunchJupyterNotInstalledInterpreter = localize(
+        'DataScience.librariesRequiredToLaunchJupyterNotInstalledInterpreter',
+        'Data Science libraries {1} are not installed in interpreter {0}.'
+    );
     export const selectJupyterInterpreter = localize('DataScience.selectJupyterInterpreter', 'Select an Interpreter to start Jupyter');
     export const jupyterInstall = localize('DataScience.jupyterInstall', 'Install');
     export const currentlySelectedJupyterInterpreterForPlaceholder = localize('Datascience.currentlySelectedJupyterInterpreterForPlaceholder', 'current: {0}');

src/client/common/utils/localize.ts

@@ -28,6 +28,8 @@ export class Activation implements IExtensionSingleActivationService {
     public async activate(): Promise<void> {
         this.disposables.push(this.notebookProvider.onDidOpenNotebookEditor(this.onDidOpenNotebookEditor, this));
         this.disposables.push(this.jupyterInterpreterService.onDidChangeInterpreter(this.onDidChangeInterpreter, this));
+        // Warm up our selected interpreter for the extension
+        this.jupyterInterpreterService.setInitialInterpreter().ignoreErrors();
         await this.contextService.activate();
     }
 

src/client/datascience/activation.ts

@@ -14,6 +14,9 @@ export const JUPYTER_OUTPUT_CHANNEL = 'JUPYTER_OUTPUT_CHANNEL';
 // Python Module to be used when instantiating the Python Daemon.
 export const PythonDaemonModule = 'datascience.jupyter_daemon';
 
+// List of 'language' names that we know about. All should be lower case as that's how we compare.
+export const KnownNotebookLanguages: string[] = ['python', 'r', 'julia', 'c++', 'c#', 'f#', 'scala', 'haskell', 'bash', 'cling', 'sas'];
+
 export namespace Commands {
     export const RunAllCells = 'python.datascience.runallcells';
     export const RunAllCellsAbove = 'python.datascience.runallcellsabove';
@@ -147,7 +150,7 @@ export enum Telemetry {
     CollapseAll = 'DATASCIENCE.COLLAPSE_ALL',
     SelectJupyterURI = 'DATASCIENCE.SELECT_JUPYTER_URI',
     SelectLocalJupyterKernel = 'DATASCIENCE.SELECT_LOCAL_JUPYTER_KERNEL',
-    SelectRemoteJupyuterKernel = 'DATASCIENCE.SELECT_REMOTE_JUPYTER_KERNEL',
+    SelectRemoteJupyterKernel = 'DATASCIENCE.SELECT_REMOTE_JUPYTER_KERNEL',
     SetJupyterURIToLocal = 'DATASCIENCE.SET_JUPYTER_URI_LOCAL',
     SetJupyterURIToUserSpecified = 'DATASCIENCE.SET_JUPYTER_URI_USER_SPECIFIED',
     Interrupt = 'DATASCIENCE.INTERRUPT',
@@ -239,7 +242,14 @@ export enum Telemetry {
     OpenedInteractiveWindow = 'DATASCIENCE.OPENED_INTERACTIVE',
     FindKernelForLocalConnection = 'DS_INTERNAL.FIND_KERNEL_FOR_LOCAL_CONNECTION',
     CompletionTimeFromLS = 'DS_INTERNAL.COMPLETION_TIME_FROM_LS',
-    CompletionTimeFromJupyter = 'DS_INTERNAL.COMPLETION_TIME_FROM_JUPYTER'
+    CompletionTimeFromJupyter = 'DS_INTERNAL.COMPLETION_TIME_FROM_JUPYTER',
+    NotebookLanguage = 'DATASCIENCE.NOTEBOOK_LANGUAGE',
+    KernelSpecNotFound = 'DS_INTERNAL.KERNEL_SPEC_NOT_FOUND',
+    KernelRegisterFailed = 'DS_INTERNAL.KERNEL_REGISTER_FAILED',
+    KernelEnumeration = 'DS_INTERNAL.KERNEL_ENUMERATION',
+    JupyterInstallFailed = 'DS_INTERNAL.JUPYTER_INSTALL_FAILED',
+    UserInstalledModule = 'DATASCIENCE.USER_INSTALLED_MODULE',
+    JupyterCommandLineNonDefault = 'DS_INTERNAL.JUPYTER_CUSTOM_COMMAND_LINE'
 }
 
 export enum NativeKeyboardCommandTelemetry {

src/client/datascience/constants.ts

@@ -24,7 +24,7 @@ import { StopWatch } from '../../common/utils/stopWatch';
 import { EXTENSION_ROOT_DIR } from '../../constants';
 import { IInterpreterService } from '../../interpreter/contracts';
 import { captureTelemetry, sendTelemetryEvent } from '../../telemetry';
-import { EditorContexts, Identifiers, NativeKeyboardCommandTelemetryLookup, NativeMouseCommandTelemetryLookup, Telemetry } from '../constants';
+import { EditorContexts, Identifiers, KnownNotebookLanguages, NativeKeyboardCommandTelemetryLookup, NativeMouseCommandTelemetryLookup, Telemetry } from '../constants';
 import { InteractiveBase } from '../interactive-common/interactiveBase';
 import {
     IEditCell,
@@ -327,7 +327,12 @@ export class NativeEditor extends InteractiveBase implements INotebookEditor {
         return this.setDirty();
     }
 
-    protected addSysInfo(_reason: SysInfoReason): Promise<void> {
+    protected addSysInfo(reason: SysInfoReason): Promise<void> {
+        // We need to send a message when restarting
+        if (reason === SysInfoReason.Restart || reason === SysInfoReason.New) {
+            this.postMessage(InteractiveWindowMessages.RestartKernel).ignoreErrors();
+        }
+
         // These are not supported.
         return Promise.resolve();
     }
@@ -398,7 +403,7 @@ export class NativeEditor extends InteractiveBase implements INotebookEditor {
         }
     }
 
-    @captureTelemetry(Telemetry.ExecuteNativeCell, undefined, false)
+    @captureTelemetry(Telemetry.ExecuteNativeCell, undefined, true)
     // tslint:disable-next-line:no-any
     protected async reexecuteCell(info: ISubmitNewCell): Promise<void> {
         try {
@@ -588,6 +593,9 @@ export class NativeEditor extends InteractiveBase implements INotebookEditor {
         // Then save the contents. We'll stick our cells back into this format when we save
         if (json) {
             this.notebookJson = json;
+
+            // Log language or kernel telemetry
+            this.sendLanguageTelemetry(this.notebookJson);
         }
         this.contentsLoadedPromise.resolve();
 
@@ -609,6 +617,27 @@ export class NativeEditor extends InteractiveBase implements INotebookEditor {
         );
     }
 
+    private sendLanguageTelemetry(notebookJson: Partial<nbformat.INotebookContent>) {
+        try {
+            // See if we have a language
+            let language = '';
+            if (notebookJson.metadata?.language_info?.name) {
+                language = notebookJson.metadata?.language_info?.name;
+            } else if (notebookJson.metadata?.kernelspec?.language) {
+                language = notebookJson.metadata?.kernelspec?.language.toString();
+            }
+            if (language && !KnownNotebookLanguages.includes(language.toLowerCase())) {
+                language = 'unknown';
+            }
+            if (language) {
+                sendTelemetryEvent(Telemetry.NotebookLanguage, undefined, { language });
+            }
+        } catch {
+            // If this fails, doesn't really matter
+            noop();
+        }
+    }
+
     private async loadCells(cells: ICell[], forceDirty: boolean): Promise<void> {
         // Make sure cells have at least 1
         if (cells.length === 0) {