Skip to content

Commit bb7522d

Browse files
AzoyAzoy
committed
[Proposal] Random Unification
Update implementation link Update current example Correctly gets a number within the bounds Fixes some typos Update implementation text Update proposal to match mailing list discussions Use `SecRandomCopyBytes` for older macOS versions Update to reflect mailing list
1 parent ebf3209 commit bb7522d

File tree

1 file changed

+397
-0
lines changed

1 file changed

+397
-0
lines changed

‎proposals/nnnn-random-unification.md

Lines changed: 397 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,397 @@
1+
# Random Unification
2+
3+
* Proposal: [SE-NNNN](NNNN-random-unification.md)
4+
* Authors: [Alejandro Alonso](https://github.com/Azoy)
5+
* Review Manager: TBD
6+
* Status: **Awaiting review**
7+
* Implementation: [apple/swift#12772](https://github.com/apple/swift/pull/12772)
8+
9+
## Introduction
10+
11+
This proposal's main focus is to create a unified random API, and a secure random API for all platforms.
12+
13+
*This idea has been floating around swift-evolution for a while now, but this is the thread that started this proposal: https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170904/039605.html*
14+
15+
## Motivation
16+
17+
The current random functionality that Swift provides is through imported C APIs. This of course is dependent on what system is running the code. This means that a different random implementation is provided on a per system basis. Swift relies on Darwin to provide developers the `arc4random(3)` approach. While on Linux, many developers tend to use `random()`, from the POSIX standard, or `rand()`, from the C standard library, for a quick and dirty solution for Linux systems. A very common implementation that developers tend to use for quick random functionality is here:
18+
19+
```swift
20+
// This is confusing because some may look at this and think Foundation provides these functions,
21+
// when in reality Foundation imports Darwin and Glibc from which these are defined and implemented.
22+
// You get the same behavior when you import UIKit, or AppKit.
23+
import Foundation
24+
25+
// We can't forget to seed the Linux implementation.
26+
// This is unsecure as time is a very predictable seed.
27+
// This raises more questions to whether or not Foundation, UIKit, or AppKit provided these functions.
28+
#if os(Linux)
29+
srandom(UInt32(time(nil)))
30+
#endif
31+
32+
// We name this randomNumber because random() interferes with Glibc's random()'s namespace
33+
func randomNumber() -> Int {
34+
#if !os(Linux)
35+
return Int(arc4random()) // This is inefficient as it doesn't utilize all of Int's range
36+
#else
37+
return random() // or Int(rand())
38+
#endif
39+
}
40+
41+
// Many tend to opt in for ranges here, but for this example I opt for a start and end argument
42+
func random(from start: Int, to end: Int) -> Int {
43+
#if !os(Linux)
44+
var random = Int(arc4random_uniform(UInt32(end - start)))
45+
#else
46+
// Not recommended as it introduces modulo bias
47+
var random = random() % (end - start)
48+
#endif
49+
50+
random += start
51+
52+
return random
53+
}
54+
55+
// Alternatively, an easier solution would be:
56+
/*
57+
func random(from start: Int, to end: Int) -> Int {
58+
var random = randomNumber() % (end - start)
59+
60+
random += start
61+
62+
return random
63+
}
64+
*/
65+
// However this approach introduces modulo bias for all systems rather than just Linux.
66+
```
67+
68+
While although this does work, it just provides a quick workaround to a much larger outstanding problem. Below is a list outlining the problems the example contains, and the problems the example introduces.
69+
70+
1. In order to define these workarounds, developers must utilize a few platform checks. Developers should not be forced to define platform checks for such trivial requests like this one.
71+
72+
2. Unexperienced developers who rely on workarounds like these, may be pushing unsecure code that is used by tens of hundreds or thousands of users. Starting with Darwin's `arc4random(3)`, pre macOS 10.12 (Sierra) and iOS 10, the implementation of `arc4random(3)` utilized the RC4 algorithm. This algorithm is now considered non-cryptographically secure due to RC4 weakness. Post macOS 10.12 (Sierra) and iOS 10, "...it was replaced with the NIST-approved AES cipher", as stated from the man pages in terminal (`man arc4random`). Moving on to Linux we see that using `random()` or `rand()` to generate numbers make it completely predictable as these weren't designed to be at a crypto level.
73+
74+
3. In the example, it uses modulo to generate the number within the upper bound. Many developers may not realize it, but for a quick workaround, it introduces modulo bias in which modulo does not correctly distribute the probability in dividing the upper bound equally within the range.
75+
76+
4. Highly inefficient as creating a new `Int` from a `UInt32` doesn't utilize the full extent of `Int`'s range.
77+
78+
## Proposed Solution
79+
80+
### Random Number Generator
81+
82+
To kick this off, we will be discussing the rngs that each operating system will utilize. We will make the default random generator cryptographically-secure. This means on each operating system, the random api for Swift will produce a secure random number. It is also worth noting that if any of the following fail, particularly reading from /dev/urandom, then we produce a fatal error and abort the application. Reasons why I went with this approach in Alternatives Considered at the bottom of this proposal.
83+
84+
#### Darwin Platform
85+
86+
1. macOS
87+
88+
| macOS < 10.12 | macOS >= 10.12 |
89+
|:--------------------------------------------------:|:-------------------:|
90+
| Use `SecRandomCopyBytes` in the Security framework | Use `arc4random(3)` |
91+
92+
2. iOS
93+
94+
| iOS < 10 | iOS >= 10 |
95+
|:--------------------------------------------------:|:-------------------:|
96+
| Use `SecRandomCopyBytes` in the Security framework | Use `arc4random(3)` |
97+
98+
#### Linux Platform
99+
100+
1. Ubuntu
101+
102+
| Kernel Version < 3.17 | Kernel Version >= 3.17 |
103+
|:----------------------:|:----------------------:|
104+
| Read from /dev/urandom | Use getrandom(2) |
105+
106+
### Random API
107+
108+
For the core API, introduce a new protocol named `RandomNumberGenerator`. This type is used to define rngs that can be used within the stdlib. Developers can conform to this type and use their own custom rng throughout their whole application.
109+
110+
Then for the stdlib's default rng implementation, introduce a new struct named `Random`. This struct contains a singleton called `default` which provides access to the methods of `RandomNumberGenerator`.
111+
112+
Finally, introduce a new protocol named `Randomizable`. This type is used to define types that have the capability of being generated randomly. A good example of this is `Color` which has a numeric range of 0x0 to 0xFFFFFF. Therefore this type has the capability of being generated randomly. In addition to being generated randomly, types that conform to `BinaryFloatingPoint` will have the capability of declaring a `Range` or `ClosedRange` to select a random value from. This functionality is also found in types that conform to `Strideable` and whose stride conforms to `SignedInteger`. They will be able to utilize `CountableRange` and `CountableClosedRange` to select a random value from.
113+
114+
Next, we will make `FixedWidthInteger`, `BinaryFloatingPoint`, and `Bool` conform to `Randomizable`. This allows for random creation of that type.
115+
116+
`FixedWidthInteger` example:
117+
```swift
118+
// The following utilizes each integer's full range extent (T.min ... T.max)
119+
120+
// Utilizes the standard library's default random (alias to Int.random(using: Random.default))
121+
let randomInt = Int.random
122+
let randomUInt = UInt.random(using: myCustomRandomNumberGenerator)
123+
124+
// The following is extended for types that conform to both Randomizable and Strideable
125+
126+
let randomIntFrom10To100 = Int.random(in: 10 ..< 100)
127+
let randomUIntFrom10Through100 = Int.random(in: 10 ... 100, using: myCustomRandomNumberGenerator)
128+
```
129+
130+
`BinaryFloatingPoint` example:
131+
```swift
132+
// For floating points, the range is set to 0 to 1 inclusive
133+
134+
// Utilizes the standard library's default random (alias to Float.random(using: Random.default))
135+
let randomFloat = Float.random
136+
let randomDouble = Double.random(using: myCustomRandomNumberGenerator)
137+
138+
// The following is extended for types that conform to both Randomizable and BinaryFloatingPoint
139+
140+
let randomFloatFrom0To5 = Float.random(in: 0.0 ..< 5.0)
141+
let randomDoubleFrom100Through103 = Double.random(in: 100 ... 103, using: myCustomRandomNumberGenerator)
142+
```
143+
144+
`Bool` example:
145+
```swift
146+
// Utilizes the standard library's default random (alias to Bool.random(using: Random.default))
147+
let randomBool1 = Bool.random
148+
let randomBool2 = Bool.random(using: myCustomRandomNumberGenerator)
149+
```
150+
151+
For `Collection` we extend these functions on that type itself. We also add a function called pick which randomly selects n amount of elements and returns those elements in an array.
152+
153+
`Collection` example:
154+
```swift
155+
let greetings = ["hey", "hi", "hello", "hola"]
156+
157+
// Utilizes the standard library's default random (alias to greetings.random(using: Random.default))
158+
print(greetings.random as Any) // This returns an Optional
159+
print(greetings.random(using: myCustomRandomNumberGenerator) as Any) // This returns an Optional
160+
161+
// The following functions are special functions for Collection
162+
163+
// Returns an array of 2 random elements
164+
print(greetings.pick(2)) // Could print ["hi", "hola"]
165+
print(greetings.pick(5, using: myCustomRandomNumberGenerator)) // If n > count, return the whole array
166+
```
167+
168+
For `Range` and `ClosedRange` we extend these types and add this functionality if the Bound is of type BinaryFloatingPoint
169+
170+
`Range` example:
171+
```swift
172+
let numbers = 0.0 ..< 5.0
173+
174+
// Utilizes the standard library's default random (alias to numbers.random(using: Random.default))
175+
print(numbers.random as Any) // This returns an Optional
176+
print(numbers.random(using: myCustomRandomNumberGenerator) as Any) // This returns an Optional
177+
```
178+
179+
`ClosedRange` example:
180+
```swift
181+
let numbers = 0.0 ... 5.0
182+
183+
// Utilizes the standard library's default random (alias to numbers.random(using: Random.default))
184+
print(numbers.random as Any) // This returns an Optional
185+
print(numbers.random(using: myCustomRandomNumberGenerator) as Any) // This returns an Optional
186+
```
187+
188+
### Shuffle API
189+
190+
As a result of adding the random api, it only makes sense to utilize that power to fuel the shuffle methods. We add a method extensions for `MutableCollection` to shuffle the collection itself, and add a method extension for `Collection` to return a shuffled version of itself in a new array. Example:
191+
192+
```swift
193+
var greetings = ["hey", "hi", "hello", "hola"]
194+
195+
// Utilizes the standard library's default random (alias to greetings.shuffle(using: Random.default))
196+
greetings.shuffle()
197+
print(greetings) // A possible output could be ["hola", "hello", "hey", "hi"]
198+
199+
let numbers = 0 ..< 5
200+
print(numbers.shuffled(using: myCustomRandomNumberGenerator)) // A possible output could be [1, 3, 0, 4, 2]
201+
```
202+
203+
## Detailed Design
204+
205+
The actual implementation can be found here: [apple/swift#12772](https://github.com/apple/swift/pull/12772)
206+
207+
```swift
208+
public protocol RandomNumberGenerator {
209+
associatedtype GeneratedNumber : FixedWidthInteger & UnsignedInteger
210+
func next() -> GeneratedNumber
211+
}
212+
213+
extension RandomNumberGenerator {
214+
public func next<T : FixedWidthInteger & UnsignedInteger>(_ type: T.Type) -> T
215+
public func next<T : FixedWidthInteger & UnsignedInteger>(_ type: T.Type, upperBound: T) -> T
216+
}
217+
218+
public struct Random : RandomNumberGenerator {
219+
public typealias GeneratedNumber = UInt
220+
221+
public static let `default` = Random()
222+
223+
// Prevents initialization of this struct
224+
private init() {}
225+
226+
// Conformance for `RandomNumberGenerator`
227+
public func next() -> UInt
228+
}
229+
230+
public protocol Collection {
231+
var random: Element? { get }
232+
func random<T: RandomNumberGenerator>(using generator: T) -> Element?
233+
}
234+
235+
extension Collection {
236+
// Default implementation
237+
public var random: Element? {
238+
return self.random(using: Random.default)
239+
}
240+
241+
// Default implementation
242+
public func random<T: RandomNumberGenerator>(using generator: T) -> Element?
243+
244+
public func pick(_ n: UInt) -> [Element]
245+
246+
public func pick<T: RandomNumberGenerator>(_ n: UInt, using generator: T) -> [Element]
247+
}
248+
249+
extension Range where Bound : BinaryFloatingPoint {
250+
public var random: Bound? {
251+
return self.random(using: Random.default)
252+
}
253+
254+
public func random<T: RandomNumberGenerator>(using generator: T) -> Bound?
255+
}
256+
257+
extension ClosedRange where Bound : BinaryFloatingPoint {
258+
public var random: Bound? {
259+
return self.random(using: Random.default)
260+
}
261+
262+
public func random<T: RandomNumberGenerator>(using generator: T) -> Bound?
263+
}
264+
265+
public protocol Randomizable {
266+
static func random<T: RandomNumberGenerator>(using generator: T) -> Self
267+
}
268+
269+
extension Randomizable {
270+
public static var random: Self {
271+
return self.random(using: Random.default)
272+
}
273+
}
274+
275+
extension Randomizable where Self: BinaryFloatingPoint {
276+
public static func random(in range: Range<Self>) -> Self {
277+
return range.random!
278+
}
279+
280+
public static func random<T: RandomNumberGenerator>(in range: Range<Self>, using generator: T) -> Self {
281+
return range.random(using: generator)!
282+
}
283+
284+
public static func random(in range: ClosedRange<Self>) -> Self {
285+
return range.random!
286+
}
287+
288+
public static func random<T: RandomNumberGenerator>(in range: ClosedRange<Self>, using generator: T) -> Self {
289+
return range.random(using: generator)!
290+
}
291+
}
292+
293+
extension Randomizable where Self: Strideable, Self.Stride: SignedInteger {
294+
public static func random(in range: CountableRange<Self>) -> Self {
295+
return range.random!
296+
}
297+
298+
public static func random<T: RandomNumberGenerator>(in range: CountableRange<Self>, using generator: T) -> Self {
299+
return range.random(using: generator)!
300+
}
301+
302+
public static func random(in range: CountableClosedRange<Self>) -> Self {
303+
return range.random!
304+
}
305+
306+
public static func random<T: RandomNumberGenerator>(in range: CountableClosedRange<Self>, using generator: T) -> Self {
307+
return range.random(using: generator)!
308+
}
309+
}
310+
311+
public protocol FixedWidthInteger : Randomizable {}
312+
313+
extension FixedWidthInteger {
314+
// Conformance to `Randomizable`
315+
public static func random<T: RandomNumberGenerator>(using generator: T) -> Self
316+
}
317+
318+
public protocol BinaryFloatingPoint : Randomizable {}
319+
320+
extension BinaryFloatingPoint {
321+
// Conformance to `Randomizable`
322+
public static func random<T: RandomNumberGenerator>(using generator: T) -> Self
323+
}
324+
325+
public struct Bool : Randomizable {}
326+
327+
extension Bool {
328+
// Conformance to `Randomizable`
329+
public static func random<T: RandomNumberGenerator>(using generator: T) -> Bool
330+
}
331+
332+
// Shuffle API
333+
334+
extension Collection {
335+
public func shuffled() -> [Element] {
336+
return self.shuffled(using: Random.default)
337+
}
338+
339+
public func shuffled<T: RandomNumberGenerator>(using generator: T) -> [Element]
340+
}
341+
342+
extension MutableCollection {
343+
public mutating func shuffle() {
344+
self.shuffle(using: Random.default)
345+
}
346+
347+
public mutating func shuffle<T: RandomNumberGenerator>(using generator: T)
348+
}
349+
```
350+
351+
Types that conform to `RandomNumberGenerator` are required to only implement the next() function. This function returns the associated type declared when conforming to this. If a generator produces 32 bit unsigned integers, then the default implementation of next(\_:) will know to call this function twice if the caller requires a 64 bit.
352+
353+
## Source compatibility
354+
355+
This change is purely additive, thus source compatibility is not affected.
356+
357+
## Effect on ABI stability
358+
359+
This change is purely additive, thus ABI stability is not affected.
360+
361+
## Effect on API resilience
362+
363+
This change is purely additive, thus API resilience is not affected.
364+
365+
## Alternatives considered
366+
367+
There were very many alternatives to be considered in this proposal.
368+
369+
### Why would the program abort if it failed to generate a random number?
370+
371+
I spent a lot of time deciding what to do if it failed. Ultimately it came down to the fact that these will almost never fail. In the cases where this can fail is where `/dev/urandom` doesn't exist, or there were too many file descriptors open on the process level or system level. In the case where `/dev/urandom` doesn't exist, either the kernel is too old to generate that file by itself on a fresh install, or a privileged user deleted it. Both of which are way out of scope for Swift in my opinion. In the case where there are too many file descriptors, with modern technology this should almost never happen. If the process has opened too many descriptors then it should be up to the developer to optimize opening and closing descriptors.
372+
373+
In a world where this did return an error to Swift, it would require types that implement `Randomizable` to return Optionals as well.
374+
375+
```swift
376+
let random = Int.random!
377+
```
378+
379+
"I just want a random number, what is this ! the compiler is telling me to add?"
380+
381+
This syntax wouldn't make sense for a custom rng that deterministically generates numbers with no fail.
382+
383+
Rust aborts when experiencing an unexpected error with any of the forms of randomness. [source](https://doc.rust-lang.org/rand/src/rand/os.rs.html)
384+
385+
It would be silly to account for these edge cases that would only happen to those who need to update their os, optimize their file descriptors, or deleted their `/dev/urandom`. Accounting for these cases sacrifices the clean api for everyone else.
386+
387+
### Shouldn't this fallback on something more secure at times of low entropy?
388+
389+
Thomas Hühn explains it very well [here](https://www.2uo.de/myths-about-urandom/). There is also a deeper discussion [here talking about python's implementation](https://www.python.org/dev/peps/pep-0524). Both articles discuss that even though /dev/urandom may not have enough entropy at a fresh install, "It doesn't matter. The underlying cryptographic building blocks are designed such that an attacker cannot predict the outcome." Using `getrandom(2)` on linux systems where the kernel version is >= 3.17, will block if it decides that the entropy pool is too small. In python's implementation, they fallback to reading `/dev/urandom` if `getrandom(2)` decides there is not enough entropy.
390+
391+
### Why not make the default rng non-secure?
392+
393+
Swift is a safe language which means that it shouldn't be encouraging non-experienced developers to be pushing unsecure code. Making the default secure removes this issue and gives developers a feeling of comfort knowing their code is secure out of the box.
394+
395+
### Rename RandomNumberGenerator
396+
397+
It has been discussed to give this a name such as `RNG`. I went with `RandomNumberGenerator` because it is concise, whereas `RNG` has a level of obscurity to those who don't know the acronym.

0 commit comments

Comments
 (0)