Skip to content

Commit 9db1a86

Browse files
gekygeky
committed
Added specification document
For covering all the technical bits
1 parent 9843402 commit 9db1a86

File tree

3 files changed

+368
-3
lines changed

3 files changed

+368
-3
lines changed

‎README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,8 @@ the littlefs was developed with the goal of learning more about filesystem
131131
design by tackling the relative unsolved problem of managing a robust
132132
filesystem resilient to power loss on devices with limited RAM and ROM.
133133
More detail on the solutions and tradeoffs incorporated into this filesystem
134-
can be found in [DESIGN.md](DESIGN.md).
134+
can be found in [DESIGN.md](DESIGN.md). The specification for the layout
135+
of the filesystem on disk can be found in [SPEC.md](SPEC.md).
135136
136137
## Testing
137138

‎SPEC.md

Lines changed: 361 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,361 @@
1+
## The little filesystem technical specification
2+
3+
This is the technical specification of the little filesystem. This document
4+
covers the technical details of how the littlefs is stored on disk for
5+
introspection and tooling development. This document assumes you are
6+
familiar with the design of the littlefs, for more info on how littlefs
7+
works check out [DESIGN.md](DESIGN.md).
8+
9+
```
10+
| | | .---._____
11+
.-----. | |
12+
--|o |---| littlefs |
13+
--| |---| |
14+
'-----' '----------'
15+
| | |
16+
```
17+
18+
## Some important details
19+
20+
- The littlefs is a block-based filesystem. This is, the disk is divided into
21+
an array of evenly sized blocks that are used as the logical unit of storage
22+
in littlefs. Block pointers are stored in 32 bits.
23+
24+
- There is no explicit free-list stored on disk, the littlefs only knows what
25+
is in use in the filesystem.
26+
27+
- The littlefs uses the value of 0xffffffff to represent a null block-pointer.
28+
29+
- All values in littlefs are stored in little-endian byte order.
30+
31+
## Directories / Metadata pairs
32+
33+
Metadata pairs form the backbone of the littlefs and provide a system for
34+
atomic updates. Even the superblock is stored in a metadata pair.
35+
36+
As their name suggests, a metadata pair is stored in two blocks, with one block
37+
acting as a redundant backup in case the other is corrupted. These two blocks
38+
could be anywhere in the disk and may not be next to each other, so any
39+
pointers to directory pairs need to be stored as two block pointers.
40+
41+
Here's the layout of metadata blocks on disk:
42+
43+
| offset | size | description |
44+
|--------|---------------|----------------|
45+
| 0x00 | 32 bits | revision count |
46+
| 0x04 | 32 bits | dir size |
47+
| 0x08 | 64 bits | tail pointer |
48+
| 0x10 | size-16 bytes | dir entries |
49+
| 0x00+s | 32 bits | crc |
50+
51+
**Revision count** - Incremented every update, only the uncorrupted
52+
metadata-block with the most recent revision count contains the valid metadata.
53+
Comparison between revision counts must use sequence comparison since the
54+
revision counts may overflow.
55+
56+
**Dir size** - Size in bytes of the contents in the current metadata block,
57+
including the metadata-pair metadata. Additionally, the highest bit of the
58+
dir size may be set to indicate that the directory's contents continue on the
59+
next metadata-pair pointed to by the tail pointer.
60+
61+
**Tail pointer** - Pointer to the next metadata-pair in the filesystem.
62+
A null pair-pointer (0xffffffff, 0xffffffff) indicates the end of the list.
63+
If the highest bit in the dir size is set, this points to the next
64+
metadata-pair in the current directory, otherwise it points to an arbitrary
65+
metadata-pair. Starting with the superblock, the tail-pointers form a
66+
linked-list containing all metadata-pairs in the filesystem.
67+
68+
**CRC** - 32 bit CRC used to detect corruption from power-lost, from block
69+
end-of-life, or just from noise on the storage bus. The CRC is appended to
70+
the end of each metadata-block. The littlefs uses the standard CRC-32, which
71+
uses a polynomial of 0x04c11db7, initialized with 0xffffffff.
72+
73+
Here's an example of a simple directory stored on disk:
74+
```
75+
(32 bits) revision count = 10 (0x0000000a)
76+
(32 bits) dir size = 154 bytes, end of dir (0x0000009a)
77+
(64 bits) tail pointer = 37, 36 (0x00000025, 0x00000024)
78+
(32 bits) crc = 0xc86e3106
79+
80+
00000000: 0a 00 00 00 9a 00 00 00 25 00 00 00 24 00 00 00 ........%...$...
81+
00000010: 22 08 00 03 05 00 00 00 04 00 00 00 74 65 61 22 "...........tea"
82+
00000020: 08 00 06 07 00 00 00 06 00 00 00 63 6f 66 66 65 ...........coffe
83+
00000030: 65 22 08 00 04 09 00 00 00 08 00 00 00 73 6f 64 e"...........sod
84+
00000040: 61 22 08 00 05 1d 00 00 00 1c 00 00 00 6d 69 6c a"...........mil
85+
00000050: 6b 31 22 08 00 05 1f 00 00 00 1e 00 00 00 6d 69 k1"...........mi
86+
00000060: 6c 6b 32 22 08 00 05 21 00 00 00 20 00 00 00 6d lk2"...!... ...m
87+
00000070: 69 6c 6b 33 22 08 00 05 23 00 00 00 22 00 00 00 ilk3"...#..."...
88+
00000080: 6d 69 6c 6b 34 22 08 00 05 25 00 00 00 24 00 00 milk4"...%...$..
89+
00000090: 00 6d 69 6c 6b 35 06 31 6e c8 .milk5.1n.
90+
```
91+
92+
A note about the tail pointer linked-list: Normally, this linked-list is
93+
threaded through the entire filesystem. However, after power-loss this
94+
linked-list may become out of sync with the rest of the filesystem.
95+
- The linked-list may contain a directory that has actually been removed
96+
- The linked-list may contain a metadata pair that has not been updated after
97+
a block in the pair has gone bad.
98+
99+
The threaded linked-list must be checked for these errors before it can be
100+
used reliably. Fortunately, the threaded linked-list can simply be ignored
101+
if littlefs is mounted read-only.
102+
103+
## Entries
104+
105+
Each metadata block contains a series of entries that follow a standard
106+
layout. An entry contains the type of the entry, along with a section for
107+
entry-specific data, attributes, and a name.
108+
109+
Here's the layout of entries on disk:
110+
111+
| offset | size | description |
112+
|---------|------------------------|----------------------------|
113+
| 0x0 | 8 bits | entry type |
114+
| 0x1 | 8 bits | entry length |
115+
| 0x2 | 8 bits | attribute length |
116+
| 0x3 | 8 bits | name length |
117+
| 0x4 | entry length bytes | entry-specific data |
118+
| 0x4+e | attribute length bytes | system-specific attributes |
119+
| 0x4+e+a | name length bytes | entry name |
120+
121+
**Entry type** - Type of the entry, currently this is limited to the following:
122+
- 0x11 - file entry
123+
- 0x22 - directory entry
124+
- 0xe2 - superblock entry
125+
126+
Additionally, the type is broken into two 4 bit nibbles, with the lower nibble
127+
specifying the type's data structure used when scanning the filesystem. The
128+
upper nibble clarifies the type further when multiple entries share the same
129+
data structure.
130+
131+
**Entry length** - Length in bytes of the entry-specific data. This does
132+
not include the entry type size, attributes, or name. The full size in bytes
133+
of the entry is 4 + entry length + attribute length + name length.
134+
135+
**Attribute length** - Length of system-specific attributes in bytes. Since
136+
attributes are system specific, there is not much garuntee on the values in
137+
this section, and systems are expected to work even when it is empty. See the
138+
[attributes](#entry-attributes) section for more details.
139+
140+
**Name length** - Length of the entry name. Entry names are stored as utf8,
141+
although most systems will probably only support ascii. Entry names can not
142+
contain '/' and can not be '.' or '..' as these are a part of the syntax of
143+
filesystem paths.
144+
145+
Here's an example of a simple entry stored on disk:
146+
```
147+
(8 bits) entry type = file (0x11)
148+
(8 bits) entry length = 8 bytes (0x08)
149+
(8 bits) attribute length = 0 bytes (0x00)
150+
(8 bits) name length = 12 bytes (0x0c)
151+
(8 bytes) entry data = 05 00 00 00 20 00 00 00
152+
(12 bytes) entry name = smallavacado
153+
154+
00000000: 11 08 00 0c 05 00 00 00 20 00 00 00 73 6d 61 6c ........ ...smal
155+
00000010: 6c 61 76 61 63 61 64 6f lavacado
156+
```
157+
158+
## Superblock
159+
160+
The superblock is the anchor for the littlefs. The superblock is stored as
161+
a metadata pair containing a single superblock entry. It is through the
162+
superblock that littlefs can access the rest of the filesystem.
163+
164+
The superblock can always be found in blocks 0 and 1, however fetching the
165+
superblock requires knowing the block size. The block size can be guessed by
166+
searching the beginning of disk for the string "littlefs", although currently
167+
the filesystems relies on the user providing the correct block size.
168+
169+
The superblock is the most valuable block in the filesystem. It is updated
170+
very rarely, only during format or when the root directory must be moved. It
171+
is encouraged to always write out both superblock pairs even though it is not
172+
required.
173+
174+
Here's the layout of the superblock entry:
175+
176+
| offset | size | description |
177+
|--------|------------------------|----------------------------------------|
178+
| 0x00 | 8 bits | entry type (0xe2 for superblock entry) |
179+
| 0x01 | 8 bits | entry length (20 bytes) |
180+
| 0x02 | 8 bits | attribute length |
181+
| 0x03 | 8 bits | name length (8 bytes) |
182+
| 0x04 | 64 bits | root directory |
183+
| 0x0c | 32 bits | block size |
184+
| 0x10 | 32 bits | block count |
185+
| 0x14 | 32 bits | version |
186+
| 0x18 | attribute length bytes | system-specific attributes |
187+
| 0x18+a | 8 bytes | magic string ("littlefs") |
188+
189+
**Root directory** - Pointer to the root directory's metadata pair.
190+
191+
**Block size** - Size of the logical block size used by the filesystem.
192+
193+
**Block count** - Number of blocks in the filesystem.
194+
195+
**Version** - The littlefs version encoded as a 32 bit value. The upper 16 bits
196+
encodes the major version, which is incremented when a breaking-change is
197+
introduced in the filesystem specification. The lower 16 bits encodes the
198+
minor version, which is incremented when a backwards-compatible change is
199+
introduced. Non-standard Attribute changes do not change the version. This
200+
specification describes version 1.1 (0x00010001), which is the first version
201+
of littlefs.
202+
203+
**Magic string** - The magic string "littlefs" takes the place of an entry
204+
name.
205+
206+
Here's an example of a complete superblock:
207+
```
208+
(32 bits) revision count = 3 (0x00000003)
209+
(32 bits) dir size = 52 bytes, end of dir (0x00000034)
210+
(64 bits) tail pointer = 3, 2 (0x00000003, 0x00000002)
211+
(8 bits) entry type = superblock (0xe2)
212+
(8 bits) entry length = 20 bytes (0x14)
213+
(8 bits) attribute length = 0 bytes (0x00)
214+
(8 bits) name length = 8 bytes (0x08)
215+
(64 bits) root directory = 3, 2 (0x00000003, 0x00000002)
216+
(32 bits) block size = 512 bytes (0x00000200)
217+
(32 bits) block count = 1024 blocks (0x00000400)
218+
(32 bits) version = 1.1 (0x00010001)
219+
(8 bytes) magic string = littlefs
220+
(32 bits) crc = 0xc50b74fa
221+
222+
00000000: 03 00 00 00 34 00 00 00 03 00 00 00 02 00 00 00 ....4...........
223+
00000010: e2 14 00 08 03 00 00 00 02 00 00 00 00 02 00 00 ................
224+
00000020: 00 04 00 00 01 00 01 00 6c 69 74 74 6c 65 66 73 ........littlefs
225+
00000030: fa 74 0b c5 .t..
226+
```
227+
228+
## Directory entries
229+
230+
Directories are stored in entries with a pointer to the first metadata pair
231+
in the directory. Keep in mind that a directory may be composed of multiple
232+
metadata pairs connected by the tail pointer when the highest bit in the dir
233+
size is set.
234+
235+
Here's the layout of a directory entry:
236+
237+
| offset | size | description |
238+
|--------|------------------------|-----------------------------------------|
239+
| 0x0 | 8 bits | entry type (0x22 for directory entries) |
240+
| 0x1 | 8 bits | entry length (8 bytes) |
241+
| 0x2 | 8 bits | attribute length |
242+
| 0x3 | 8 bits | name length |
243+
| 0x4 | 64 bits | directory pointer |
244+
| 0xc | attribute length bytes | system-specific attributes |
245+
| 0xc+a | name length bytes | directory name |
246+
247+
**Directory pointer** - Pointer to the first metadata pair in the directory.
248+
249+
Here's an example of a directory entry:
250+
```
251+
(8 bits) entry type = directory (0x22)
252+
(8 bits) entry length = 8 bytes (0x08)
253+
(8 bits) attribute length = 0 bytes (0x00)
254+
(8 bits) name length = 3 bytes (0x03)
255+
(64 bits) directory pointer = 5, 4 (0x00000005, 0x00000004)
256+
(3 bytes) name = tea
257+
258+
00000000: 22 08 00 03 05 00 00 00 04 00 00 00 74 65 61 "...........tea
259+
```
260+
261+
## File entries
262+
263+
Files are stored in entries with a pointer to the head of the file and the
264+
size of the file. This is enough information to determine the state of the
265+
CTZ linked-list that is being referenced.
266+
267+
How files are actually stored on disk is a bit complicated. The full
268+
explanation of CTZ linked-lists can be found in [DESIGN.md](DESIGN.md#ctz-linked-lists).
269+
270+
A terribly quick summary: For every nth block where n is divisible by 2^x,
271+
the block contains a pointer that points x blocks towards the beginning of the
272+
file. These pointers are stored in order of x in each block of the file
273+
immediately before the data in the block.
274+
275+
Here's the layout of a file entry:
276+
277+
| offset | size | description |
278+
|--------|------------------------|------------------------------------|
279+
| 0x0 | 8 bits | entry type (0x11 for file entries) |
280+
| 0x1 | 8 bits | entry length (8 bytes) |
281+
| 0x2 | 8 bits | attribute length |
282+
| 0x3 | 8 bits | name length |
283+
| 0x4 | 32 bits | file head |
284+
| 0x8 | 32 bits | file size |
285+
| 0xc | attribute length bytes | system-specific attributes |
286+
| 0xc+a | name length bytes | directory name |
287+
288+
**File head** - Pointer to the block that is the head of the file's CTZ
289+
linked-list.
290+
291+
**File size** - Size of file in bytes.
292+
293+
Here's an example of a file entry:
294+
```
295+
(8 bits) entry type = file (0x11)
296+
(8 bits) entry length = 8 bytes (0x08)
297+
(8 bits) attribute length = 0 bytes (0x00)
298+
(8 bits) name length = 12 bytes (0x03)
299+
(32 bits) file head = 543 (0x0000021f)
300+
(32 bits) file size = 256 KB (0x00040000)
301+
(12 bytes) name = largeavacado
302+
303+
00000000: 11 08 00 0c 1f 02 00 00 00 00 04 00 6c 61 72 67 ............larg
304+
00000010: 65 61 76 61 63 61 64 6f eavacado
305+
```
306+
307+
## Entry attributes
308+
309+
Each dir entry can have up to 256 bytes of system-specific attributes. Since
310+
these attributes are system-specific, they may not be portable between
311+
different systems. For this reason, all attributes must be optional. A minimal
312+
littlefs driver must be able to get away with supporting no attributes at all.
313+
314+
For some level of portability, littlefs has a simple scheme for attributes.
315+
Each attribute is prefixes with an 8-bit type that indicates what the attribute
316+
is. The length of attributes may also be determined from this type. Attributes
317+
in an entry should be sorted based on portability, since attribute parsing
318+
will end when it hits the first attribute it does not understand.
319+
320+
Each system should choose a 4-bit value to prefix all attribute types with to
321+
avoid conflicts with other systems. Additionally, littlefs drivers that support
322+
attributes should provide a "ignore attributes" flag to users in case attribute
323+
conflicts do occur.
324+
325+
Attribute types prefixes with 0x0 and 0xf are currently reserved for future
326+
standard attributes. Standard attributes will be added to this document in
327+
that case.
328+
329+
Here's an example of non-standard time attribute:
330+
```
331+
(8 bits) attribute type = time (0xc1)
332+
(72 bits) time in seconds = 1506286115 (0x0059c81a23)
333+
334+
00000000: c1 23 1a c8 59 00 .#..Y.
335+
```
336+
337+
Here's an example of non-standard permissions attribute:
338+
```
339+
(8 bits) attribute type = permissions (0xc2)
340+
(16 bits) permission bits = rw-rw-r-- (0x01b4)
341+
342+
00000000: c2 b4 01 ...
343+
```
344+
345+
Here's what a dir entry may look like with these attributes:
346+
```
347+
(8 bits) entry type = file (0x11)
348+
(8 bits) entry length = 8 bytes (0x08)
349+
(8 bits) attribute length = 9 bytes (0x09)
350+
(8 bits) name length = 12 bytes (0x0c)
351+
(8 bytes) entry data = 05 00 00 00 20 00 00 00
352+
(8 bits) attribute type = time (0xc1)
353+
(72 bits) time in seconds = 1506286115 (0x0059c81a23)
354+
(8 bits) attribute type = permissions (0xc2)
355+
(16 bits) permission bits = rw-rw-r-- (0x01b4)
356+
(12 bytes) entry name = smallavacado
357+
358+
00000000: 11 08 09 0c 05 00 00 00 20 00 00 00 c1 23 1a c8 ........ ....#..
359+
00000010: 59 00 c2 b4 01 73 6d 61 6c 6c 61 76 61 63 61 64 Y....smallavacad
360+
00000020: 6f o
361+
```

‎lfs_util.h

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@
1212
#include <stdio.h>
1313

1414

15-
// Builtin functions
15+
// Builtin functions, these may be replaced by more
16+
// efficient implementations in the system
1617
static inline uint32_t lfs_max(uint32_t a, uint32_t b) {
1718
return (a > b) ? a : b;
1819
}
@@ -33,10 +34,12 @@ static inline int lfs_scmp(uint32_t a, uint32_t b) {
3334
return (int)(unsigned)(a - b);
3435
}
3536

37+
// CRC-32 with polynomial = 0x04c11db7
3638
void lfs_crc(uint32_t *crc, const void *buffer, size_t size);
3739

3840

39-
// Logging functions
41+
// Logging functions, these may be replaced by system-specific
42+
// logging functions
4043
#define LFS_DEBUG(fmt, ...) printf("lfs debug: " fmt "\n", __VA_ARGS__)
4144
#define LFS_WARN(fmt, ...) printf("lfs warn: " fmt "\n", __VA_ARGS__)
4245
#define LFS_ERROR(fmt, ...) printf("lfs error: " fmt "\n", __VA_ARGS__)

0 commit comments

Comments
 (0)