Download MNIST dataset from remote URL #215

spencerkohan · 2019-10-10T15:50:10Z

This PR is intended to fix the second item listed in this issue:

it currently relies on hardcoded paths, which will break if you take the dataset outside of that project and try to use it in something like Colab.

In this PR, the datasets are instead downloaded from a remote URL, so they are not fragile with respect to the current working directory where the MNIST dataset is instantiated from.

The implementation in this PR brings the MINST dataset in line with how the CFAR10 dataset is loaded.

…than relying on local paths

BradLarson · 2019-10-10T23:06:00Z

Datasets/DatasetUtils.swift

+        } catch {
+            fatalError("Failed to contents of resource: \(localURL)")
+        }
+


Minor style thing, but I don't think we need the extra spaces inserted after many of the closing (and some opening) braces here.

BradLarson

Thanks for taking this on and fixing the MNIST download process. Sorry I didn't get to it earlier.

Datasets/DatasetUtils.swift

rxwei · 2019-10-10T23:11:31Z

Datasets/DatasetUtils.swift

+
+        let localURL = resource.localURL
+
+        // Fetch from remote if the file is not available locally


Use three slashes for doc comments.

Start a doc comment with a third-person singular verb.

End a doc comment with a period.

This comment seemed redundant with the print statement, and it's pretty self-explanatory from the code anyway, so I just removed it. Let me know if you think it's better to have a doc comment there.

Datasets/DatasetUtils.swift

Datasets/MNIST/MNIST.swift

Datasets/DatasetUtils.swift

Datasets/MNIST/MNIST.swift

Co-Authored-By: Richard Wei <[email protected]>

…erkohan/swift-models into download-mnist-from-remote

spencerkohan · 2019-10-12T17:21:07Z

@BradLarson @rxwei I believe I have made all the requested changes. Thanks for taking the time to review.

rxwei · 2019-10-15T16:38:08Z

Datasets/DatasetUtilities.swift

+        }
+    }
+
+    static func fetchFromRemoteAndSave(resource: ResourceDefinition) {


The resource: argument label is seemingly only repeating the type information ResourceDefinition, and it doesn't affect the grammaticality of the phrase at call sites. I'd suggest dropping the argument label.

Suggested change

static func fetchFromRemoteAndSave(resource: ResourceDefinition) {

static func fetchFromRemoteAndSave(_ resource: ResourceDefinition) {

* Use PascalCase for all Swift source files (#152) Except `main.swift` files which are the entry points for executables. Running the following command does not produce any output: `find . -type f -name "[[:lower:]]*.swift" | rg --pcre2 "^(?!.*(main.swift))"` Resolves #133. * Load config variable from `hparams.json` file so that Transformer can work with the bigger GPT-2 models (It is called "staged release". as of now, only 117M and 345M are available). (#154) * Style fixes from #154 (#155) * A few quick fixes to help unbreak swift-models. (#160) * Unbreak models by inserting `.call`. (#161) See https://bugs.swift.org/browse/TF-516 for additional context. * s/CotangentVector/TangentVector/g (#162) * Add MNIST test set evaluation and change hyperparameters. (#163) - Add MNIST test set evaluation. - The batch size and the optimizer are changed to `128` and `Adam`, respectively, based on test set evaluation results. Tested with higher batch sizes [256, 512], but found no improvement in performance. * Updated models to use revised name for callable (SE-0253), fixed a few issues caused by `swift-apis` changes. (#166) * Updated Transformer and MNIST models to work with 2019-06-04 development snapshot. * Updated other models. * Updated ResNet to work with 2019-06-04 development snapshot. * Updated Catch and CIFAR models. * Added a method to satisfy TensorGroup protocol that can pass build. * transformer: upstream api changes (#171) * rebuild resnet block based approach (#170) * add gym blackjack qlearning demo (#173) * Make 'GoModel' stored properties be variables. (#174) New 'Differentiable' derived conformances will not include constant properties in the tangent space. * Added .swift-format and updated .gitignore for Xcode 11's SwiftPM support. (#176) This replicates the following pull request on swift-apis: tensorflow/swift-apis#374 , adding a `swift-format` configuration file. Note the cautions indicated in that pull request around use of `swift-format` at present. This also adds a .gitignore line to prevent Xcode 11's new SwiftPM support from adding supporting files to the repository. * Replaced Python with Swift in CIFAR10 dataset loading (#178) * Replaced all Python code in the CIFAR10 and ResNet examples, removing Python 3 dependency. * Needed to import FoundationNetworking on Linux. * Added a check for FoundationNetworking, added an early exit for cached directory check. * Removed macOS availability check by targeting 10.13 in the package. * Style and formatting fixes. * Removed no-longer-needed _tensorHandles and supporting code. * Replace Autoencoder's tanh activation with sigmoid (#180) * Add GAN Example (#181) * Add GAN Example * Remove do blocks * Replace lrelu with closure * Remove typealiases * Add comment * Remove labels in GAN example * Rename variables * Code formatting, update comments * Fix: latentDim -> latentSize * Add spaces around * * Code formatting, rename loss functions * Fix: generatorLossFunc -> generatorLoss * Break lines to make it fit within 100 columns * Update comments * Rename plot->plotImage, imageGrid->gridImage, Code formatting * Refactor: Label creation * Update comment * Remove type parameter, empty line * Update readme * nightlies URL: s4tf-kokoro-artifact-testing => swift-tensorflow-artifacts (#183) * First steps in repository reorganization: extracting common MNIST dataset code (#182) * Extracted MNIST dataset, created LeNet network, added example combining the two. * Extracted redundant MNIST loading code from GAN and Autoencoder examples, replaced with central MNIST dataset. * Renamed input parameters and applied standard formatting style to MNIST. * Punctuation correction. Co-Authored-By: Richard Wei <[email protected]> * README formatting update. Co-Authored-By: Richard Wei <[email protected]> * Renamed trainImages -> trainingImages, corrected Python package names, formatted Package.swift. * Update Dockerfile to install Python libraries (#184) * Delete helpers.swift (#187) This file is no longer needed apple/swift #26023 and may be causing the segfault reported in swift-apis #186. * Continuing repository reorganization: extracting common CIFAR-10 and ResNet code (#185) * Extracted CIFAR-10 dataset and ResNet models into respective modules. * Minor formatting fixes. * Mirroring PR #187. * [Models] Fix enumeration of blocks in `ResidualBasicBlockStack` `init` (#192) * [Models] Fix enumeration of blocks in `ResidualBasicBlockStack` initialization * Start range at * Update ranges * SqueezeNet Implementation (#189) * Add files via upload * Implemented requested changes Corrected formatting mistakes, used dropout correctly, removed the hard coded number of classes and improved overall readability. * Update SqeezeNet.swift Renamed numClasses to classCount and limited all lines to a size of 100. * WideResNet - fix widenFactor and match model to citation (#193) * add identity connections to WideResNet * rename preact1 for clarity * remove extra relu, add dropout * fix declarartion * skip dropout in expansion blocks * remove enum, res layers to one line * Removal of deprecated allDifferentiableVariables. (#194) * Add JPEG loading / saving via Raw TF operations, removing Matplotlib dependencies (#188) * Added an Image struct as a wrapper for JPEG loading / saving, removed matplotlib dependency from GAN and Autoencoder examples using this. * Formatting update for Autoencoder. * Bring this inline with current API. * Made saveImage() a throwing function, improved formatting. * Changed function parameter. * Convert MNIST classifier to use sequential (#200) * Convert MNIST classifier to use sequential * Review Changes * Remove ImageClassification Models * Convert Autoencoder and Catch to use Sequential (#203) Partially fixes #202. * Adding shape-based inference tests for all image classification models (#198) * Implemented inference tests with random tensors and starting weights for all classification models. * Made sure tests ran on Linux, reshaped output of SqueezeNet to match other classification models. * WideResNet is expressed in terms of CIFAR10, so altered the inputs and outputs appropriately. * Wrapping the reshaping line in SqueezeNet. * Reset ownership. * Reworked TensorShape initializers to use array literals. * Minor formatting tweak to SqueezeNet. * Update deprecated APIs in Catch (#205) [swift-apis/#497](tensorflow/swift-apis#497) removes deprecated apis, now that we are moving onward to v0.5. This PR updates the catch file which uses a deprecated dense layer. * Update squeezenet and add V1.1 version (#204) * Update squeezenet and add V1.1 version * value error * Update tests * review changes * Fix '@noDerivative' warnings. (#208) * Indentation and formatting fixes in CartPole (#214) * DenseNet Implementaion (#213) * Create DenseNet.swift Implementation for the DenseNet Architecture * Added test for DenseNet * Fixed Typo * Refractored the code * Updated ConvPair description Co-Authored-By: Richard Wei <[email protected]> * Fixed comment Co-Authored-By: Richard Wei <[email protected]> * Fixed typos, and improved code quality * Removed extra whitespace and '\' * Used 3 '/' for documentation and removed return * Renamed DenseNet to DenseNet121 * Renamed tests * Updated names * Renamed all DenseNet identifiers to DenseNet121 * Download MNIST dataset from remote URL (#215) This PR is intended to fix the second item listed in [this issue](#206): > it currently relies on hardcoded paths, which will break if you take the dataset outside of that project and try to use it in something like Colab. In this PR, the datasets are instead downloaded from a remote URL, so they are not fragile with respect to the current working directory where the MNIST dataset is instantiated from. The implementation in this PR brings the MINST dataset in line with how the CFAR10 dataset is loaded. * use enable in _vjpCausallyMasked (#218) Fixes #217 * Change momentum and epsilon properties to scalars. (#220) * Uses switch instead of ifs (#219) * Fixes deprecation warning in MiniGo (#221) * Fixing deprecation warnings on all but the MiniGo example. (#223) * Initial training and inference benchmarks (#222) * Initial implementation of training and inference benchmarks. * Resolved a typechecker issue. * Added a Readme. * Added copyright headers. * Reworked the benchmarking functions to use a single function that is specialized via enums. * Reworked the model-specific benchmark into general inference and training cases that can work on arbitrary image classification models. * Restructured BenchmarkResults to be a plain collection of timings, with specific benchmark processing working on that. * Rename interpretTimings to interpretedTimings. Co-Authored-By: Dan Zheng <[email protected]> * Change reference to interpretedTimings later.

spencerkohan added 4 commits October 10, 2019 17:31

modified MINST dataset instantiation to download the datasets rather …

7b9e76f

…than relying on local paths

made import of foundation networking conditional in DatasetUtils.swift

1980cc1

replaced extension removed in error

2238587

ran swift-format to conform to style guidelines

0fba55c

BradLarson reviewed Oct 10, 2019

View reviewed changes

BradLarson approved these changes Oct 10, 2019

View reviewed changes

rxwei suggested changes Oct 10, 2019

View reviewed changes

Datasets/DatasetUtils.swift Outdated Show resolved Hide resolved

Datasets/MNIST/MNIST.swift Outdated Show resolved Hide resolved

Datasets/MNIST/MNIST.swift Outdated Show resolved Hide resolved

spencerkohan and others added 4 commits October 11, 2019 10:40

Update Datasets/DatasetUtils.swift

df47928

Co-Authored-By: Richard Wei <[email protected]>

Update Datasets/DatasetUtils.swift

0ffee67

Co-Authored-By: Richard Wei <[email protected]>

formatting fixes

215adf2

Merge branch 'download-mnist-from-remote' of https://github.com/spenc…

776b7a2

…erkohan/swift-models into download-mnist-from-remote

spencerkohan requested a review from rxwei October 15, 2019 10:05

rxwei reviewed Oct 15, 2019

View reviewed changes

removed redundant argument name

f503150

rxwei approved these changes Oct 15, 2019

View reviewed changes

rxwei added the kokoro:run label Oct 15, 2019

kokoro-team removed the kokoro:run label Oct 15, 2019

fixed error with renamed struct

328854a

rxwei added the kokoro:run label Oct 15, 2019

kokoro-team removed the kokoro:run label Oct 15, 2019

rxwei added kokoro:run kokoro:force-run labels Oct 15, 2019

kokoro-team removed kokoro:run kokoro:force-run labels Oct 15, 2019

rxwei merged commit 41c4a7a into tensorflow:master Oct 15, 2019

huan mentioned this pull request Oct 15, 2019

Some known problems of the MNIST implmentation #206

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Download MNIST dataset from remote URL #215

Download MNIST dataset from remote URL #215

Uh oh!

spencerkohan commented Oct 10, 2019

Uh oh!

BradLarson Oct 10, 2019

Uh oh!

BradLarson left a comment

Uh oh!

Uh oh!

Uh oh!

rxwei Oct 10, 2019

Uh oh!

spencerkohan Oct 12, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spencerkohan commented Oct 12, 2019

Uh oh!

rxwei Oct 15, 2019

Uh oh!

Uh oh!


		let localURL = resource.localURL

		// Fetch from remote if the file is not available locally

	static func fetchFromRemoteAndSave(resource: ResourceDefinition) {
	static func fetchFromRemoteAndSave(_ resource: ResourceDefinition) {

Download MNIST dataset from remote URL #215

Download MNIST dataset from remote URL #215

Uh oh!

Conversation

spencerkohan commented Oct 10, 2019

Uh oh!

BradLarson Oct 10, 2019

Choose a reason for hiding this comment

Uh oh!

BradLarson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rxwei Oct 10, 2019

Choose a reason for hiding this comment

Uh oh!

spencerkohan Oct 12, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spencerkohan commented Oct 12, 2019

Uh oh!

rxwei Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!