API Reference

User Functions

fsspec.open_files(urlpath[, mode, ...])	Given a path or paths, return a list of OpenFile objects.
fsspec.open(urlpath[, mode, compression, ...])	Given a path or paths, return one OpenFile object.
fsspec.open_local(url[, mode])	Open file(s) which can be resolved to local
fsspec.filesystem(protocol, **storage_options)	Instantiate filesystems for given protocol and arguments
fsspec.get_filesystem_class(protocol)	Fetch named protocol implementation from the registry
fsspec.get_mapper(url[, check, create, ...])	Create key-value interface for given URL and options
fsspec.fuse.run(fs, path, mount_point[, ...])	Mount stuff in a local directory
fsspec.gui.FileSelector([url, filters, ...])	Panel-based graphical file selector widget

fsspec.open_files(urlpath, mode='rb', compression=None, encoding='utf8', errors=None, name_function=None, num=1, protocol=None, newline=None, auto_mkdir=True, expand=True, **kwargs)

Given a path or paths, return a list of OpenFile objects.

For writing, a str path must contain the “*” character, which will be filled in by increasing numbers, e.g., “part*” -> “part1”, “part2” if num=2.

For either reading or writing, can instead provide explicit list of paths.

Parameters

urlpath: string or list

Absolute or relative filepath(s). Prefix with a protocol like s3:// to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol.

mode: ‘rb’, ‘wt’, etc.

compression: string or None

If given, open file using compression codec. Can either be a compression name (a key in fsspec.compression.compr) or “infer” to guess the compression from the filename suffix.

encoding: str

For text mode only

errors: None or str

Passed to TextIOWrapper in text mode

name_function: function or None

if opening a set of files for writing, those files do not yet exist, so we need to generate their names by formatting the urlpath for each sequence number

num: int [1]

if writing mode, number of files we expect to create (passed to name+function)

protocol: str or None

If given, overrides the protocol found in the URL.

newline: bytes or None

Used for line terminator in text mode. If None, uses system default; if blank, uses no translation.

auto_mkdir: bool (True)

If in write mode, this will ensure the target directory exists before writing, by calling fs.mkdirs(exist_ok=True).

expand: bool

**kwargs: dict

Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc.

Returns

An OpenFiles instance, which is a list of OpenFile objects that can

be used as a single context

Examples

>>> files = open_files('2015-*-*.csv')  
>>> files = open_files(
...     's3://bucket/2015-*-*.csv.gz', compression='gzip'
... )  

fsspec.open(urlpath, mode='rb', compression=None, encoding='utf8', errors=None, protocol=None, newline=None, **kwargs)

Given a path or paths, return one OpenFile object.

Parameters

urlpath: string or list

Absolute or relative filepath. Prefix with a protocol like s3:// to read from alternative filesystems. Should not include glob character(s).

mode: ‘rb’, ‘wt’, etc.

compression: string or None

If given, open file using compression codec. Can either be a compression name (a key in fsspec.compression.compr) or “infer” to guess the compression from the filename suffix.

encoding: str

For text mode only

errors: None or str

Passed to TextIOWrapper in text mode

protocol: str or None

If given, overrides the protocol found in the URL.

newline: bytes or None

Used for line terminator in text mode. If None, uses system default; if blank, uses no translation.

**kwargs: dict

Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc.

Returns

OpenFile object.

Examples

>>> openfile = open('2015-01-01.csv')  
>>> openfile = open(
...     's3://bucket/2015-01-01.csv.gz', compression='gzip'
... )  
>>> with openfile as f:
...     df = pd.read_csv(f)  
...

fsspec.open_local(url, mode='rb', **storage_options)

Open file(s) which can be resolved to local

For files which either are local, or get downloaded upon open (e.g., by file caching)

Parameters

url: str or list(str)

mode: str

Must be read mode

storage_options:

passed on to FS for or used by open_files (e.g., compression)

fsspec.filesystem(protocol, **storage_options)

Instantiate filesystems for given protocol and arguments

storage_options are specific to the protocol being chosen, and are passed directly to the class.

fsspec.get_filesystem_class(protocol)

Fetch named protocol implementation from the registry

The dict known_implementations maps protocol names to the locations of classes implementing the corresponding file-system. When used for the first time, appropriate imports will happen and the class will be placed in the registry. All subsequent calls will fetch directly from the registry.

Some protocol implementations require additional dependencies, and so the import may fail. In this case, the string in the “err” field of the known_implementations will be given as the error message.

fsspec.get_mapper(url, check=False, create=False, missing_exceptions=None, alternate_root=None, **kwargs)

Create key-value interface for given URL and options

The URL will be of the form “protocol://location” and point to the root of the mapper required. All keys will be file-names below this location, and their values the contents of each key.

Also accepts compound URLs like zip::s3://bucket/file.zip , see fsspec.open.

Parameters

url: str

Root URL of mapping

check: bool

Whether to attempt to read from the location before instantiation, to check that the mapping does exist

create: bool

Whether to make the directory corresponding to the root before instantiating

missing_exceptions: None or tuple

If given, these exception types will be regarded as missing keys and return KeyError when trying to read data. By default, you get (FileNotFoundError, IsADirectoryError, NotADirectoryError)

alternate_root: None or str

In cases of complex URLs, the parser may fail to pick the correct part for the mapper root, so this arg can override

Returns

FSMap instance, the dict-like key-value store.

fsspec.fuse.run(fs, path, mount_point, foreground=True, threads=False, ready_file=False, ops_class=<class 'fsspec.fuse.FUSEr'>)

Mount stuff in a local directory

This uses fusepy to make it appear as if a given path on an fsspec instance is in fact resident within the local file-system.

This requires that fusepy by installed, and that FUSE be available on the system (typically requiring a package to be installed with apt, yum, brew, etc.).

Parameters

fs: file-system instance

From one of the compatible implementations

path: str

Location on that file-system to regard as the root directory to mount. Note that you typically should include the terminating “/” character.

mount_point: str

An empty directory on the local file-system where the contents of the remote path will appear.

foreground: bool

Whether or not calling this function will block. Operation will typically be more stable if True.

threads: bool

Whether or not to create threads when responding to file operations within the mounter directory. Operation will typically be more stable if False.

ready_file: bool

Whether the FUSE process is ready. The fuse_ready file will exist in the mount_point directory if True. Debugging purpose.

ops_class: FUSEr or Subclass of FUSEr

To override the default behavior of FUSEr. For Example, logging to file.

class fsspec.gui.FileSelector(url=None, filters=None, ignore=None, kwargs=None)

Panel-based graphical file selector widget

Instances of this widget are interactive and can be displayed in jupyter by having them as the output of a cell, or in a separate browser tab using .show().

Attributes

fs

Current filesystem instance

storage_options

Value of the kwargs box as a dictionary

urlpath

URL of currently selected item

Methods

connect(signal, slot)	Associate call back with given event
ignore_events()	Temporarily turn off events processing in this instance
open_file([mode, compression, encoding])	Create OpenFile instance for the currently selected item
show([threads])	Open a new browser tab and display this instance's interface

filters_changed
go_clicked
home_clicked
protocol_changed
selection_changed
set_filters
up_clicked

property fs

Current filesystem instance

open_file(mode='rb', compression=None, encoding=None)

Create OpenFile instance for the currently selected item

For example, in a notebook you might do something like

[ ]: sel = FileSelector(); sel

# user selects their file

[ ]: with sel.open_file('rb') as f:
...      out = f.read()

Parameters

mode: str (optional)

Open mode for the file.

compression: str (optional)

The interact with the file as compressed. Set to ‘infer’ to guess compression from the file ending

encoding: str (optional)

If using text mode, use this encoding; defaults to UTF8.

property storage_options

Value of the kwargs box as a dictionary

property urlpath

URL of currently selected item

Base Classes

fsspec.spec.AbstractFileSystem(*args, **kwargs)	An abstract super-class for pythonic file-systems
fsspec.spec.Transaction(fs)	Filesystem transaction write context
fsspec.spec.AbstractBufferedFile(fs, path[, ...])	Convenient class to derive from to provide buffering
fsspec.archive.AbstractArchiveFileSystem(...)	A generic superclass for implementing Archive-based filesystems.
fsspec.FSMap(root, fs[, check, create, ...])	Wrap a FileSystem instance as a mutable wrapping.
fsspec.asyn.AsyncFileSystem(*args, **kwargs)	Async file operations, default implementations
fsspec.core.OpenFile(fs, path[, mode, ...])	File-like object to be used in a context
fsspec.core.OpenFiles(*args[, mode, fs])	List of OpenFile instances
fsspec.core.BaseCache(blocksize, fetcher, size)	Pass-though cache: doesn't keep anything, calls every time
fsspec.core.get_fs_token_paths(urlpath[, ...])	Filesystem, deterministic token, and paths from a urlpath and options.
fsspec.core.url_to_fs(url, **kwargs)	Turn fully-qualified and potentially chained URL into filesystem instance
fsspec.dircache.DirCache([...])	Caching of directory listings, in a structure like.
fsspec.registry.ReadOnlyRegistry(target)	Dict-like registry, but immutable
fsspec.registry.register_implementation(...)	Add implementation class to the registry
fsspec.callbacks.Callback([size, value, hooks])	Base class and interface for callback mechanism
fsspec.callbacks.NoOpCallback([size, value, ...])	This implementation of Callback does exactly nothing
fsspec.callbacks.DotPrinterCallback([...])	Simple example Callback implementation

class fsspec.spec.AbstractFileSystem(*args, **kwargs)

An abstract super-class for pythonic file-systems

Implementations are expected to be compatible with or, better, subclass from here.

Attributes

transaction

A context within which files are committed together upon exit

Methods

cat(path[, recursive, on_error])	Fetch (potentially multiple) paths' contents
cat_file(path[, start, end])	Get the content of a file
checksum(path)	Unique value for current version of file
clear_instance_cache()	Clear the cache of filesystem instances.
copy(path1, path2[, recursive, on_error])	Copy within two locations in the filesystem
cp(path1, path2, **kwargs)	Alias of AbstractFileSystem.copy.
created(path)	Return the created timestamp of a file as a datetime.datetime
current()	Return the most recently created FileSystem
delete(path[, recursive, maxdepth])	Alias of AbstractFileSystem.rm.
disk_usage(path[, total, maxdepth])	Alias of AbstractFileSystem.du.
download(rpath, lpath[, recursive])	Alias of AbstractFileSystem.get.
du(path[, total, maxdepth])	Space used by files within a path
end_transaction()	Finish write transaction, non-context version
exists(path, **kwargs)	Is there a file at the given path
expand_path(path[, recursive, maxdepth])	Turn one or more globs or directories into a list of all matching paths to files or directories.
find(path[, maxdepth, withdirs])	List all files below path.
from_json(blob)	Recreate a filesystem instance from JSON representation
get(rpath, lpath[, recursive, callback])	Copy file(s) to local.
get_file(rpath, lpath[, callback])	Copy single remote file to local
get_mapper(root[, check, create])	Create key/value store based on this file-system
glob(path, **kwargs)	Find files by glob-matching.
head(path[, size])	Get the first size bytes from file
info(path, **kwargs)	Give details of entry at path
invalidate_cache([path])	Discard any cached directory information
isdir(path)	Is this entry directory-like?
isfile(path)	Is this entry file-like?
lexists(path, **kwargs)	If there is a file at the given path (including broken links)
listdir(path[, detail])	Alias of AbstractFileSystem.ls.
ls(path[, detail])	List objects at path.
makedir(path[, create_parents])	Alias of AbstractFileSystem.mkdir.
makedirs(path[, exist_ok])	Recursively make directories
mkdir(path[, create_parents])	Create directory entry at path
mkdirs(path[, exist_ok])	Alias of AbstractFileSystem.makedirs.
modified(path)	Return the modified timestamp of a file as a datetime.datetime
move(path1, path2, **kwargs)	Alias of AbstractFileSystem.mv.
mv(path1, path2[, recursive, maxdepth])	Move file(s) from one location to another
open(path[, mode, block_size, ...])	Return a file-like object from the filesystem
pipe(path[, value])	Put value into path
pipe_file(path, value, **kwargs)	Set the bytes of given file
put(lpath, rpath[, recursive, callback])	Copy file(s) from local.
put_file(lpath, rpath[, callback])	Copy single file to remote
read_block(fn, offset, length[, delimiter])	Read a block of bytes from
rename(path1, path2, **kwargs)	Alias of AbstractFileSystem.mv.
rm(path[, recursive, maxdepth])	Delete files.
rm_file(path)	Delete a file
rmdir(path)	Remove a directory, if empty
sign(path[, expiration])	Create a signed URL representing the given path
size(path)	Size in bytes of file
sizes(paths)	Size in bytes of each file in a list of paths
start_transaction()	Begin write transaction for deferring files, non-context version
stat(path, **kwargs)	Alias of AbstractFileSystem.info.
tail(path[, size])	Get the last size bytes from file
to_json()	JSON representation of this filesystem instance
touch(path[, truncate])	Create empty file, or update timestamp
ukey(path)	Hash of file properties, to tell if it has changed
upload(lpath, rpath[, recursive])	Alias of AbstractFileSystem.put.
walk(path[, maxdepth])	Return all files belows path

cat_ranges
cp_file

cat(path, recursive=False, on_error='raise', **kwargs)

Fetch (potentially multiple) paths’ contents

Parameters

recursive: bool

If True, assume the path(s) are directories, and get all the contained files

on_error“raise”, “omit”, “return”

If raise, an underlying exception will be raised (converted to KeyError if the type is in self.missing_exceptions); if omit, keys with exception will simply not be included in the output; if “return”, all keys are included in the output, but the value will be bytes or an exception instance.

kwargs: passed to cat_file

Returns

dict of {path: contents} if there are multiple paths

or the path has been otherwise expanded

cat_file(path, start=None, end=None, **kwargs)

Get the content of a file

Parameters

path: URL of file on this filesystems

start, end: int

Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

kwargs: passed to ``open()``.

checksum(path)

Unique value for current version of file

If the checksum is the same from one moment to another, the contents are guaranteed to be the same. If the checksum changes, the contents might have changed.

This should normally be overridden; default will probably capture creation/modification timestamp (which would be good) or maybe access timestamp (which would be bad)

classmethod clear_instance_cache()

Clear the cache of filesystem instances.

Notes

Unless overridden by setting the cachable class attribute to False, the filesystem class stores a reference to newly created instances. This prevents Python’s normal rules around garbage collection from working, since the instances refcount will not drop to zero until clear_instance_cache is called.

copy(path1, path2, recursive=False, on_error=None, **kwargs)

Copy within two locations in the filesystem

on_error“raise”, “ignore”

If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp(path1, path2, **kwargs)

Alias of AbstractFileSystem.copy.

created(path)

Return the created timestamp of a file as a datetime.datetime

classmethod current()

Return the most recently created FileSystem

If no instance has been created, then create one with defaults

delete(path, recursive=False, maxdepth=None)

Alias of AbstractFileSystem.rm.

disk_usage(path, total=True, maxdepth=None, **kwargs)

Alias of AbstractFileSystem.du.

download(rpath, lpath, recursive=False, **kwargs)

Alias of AbstractFileSystem.get.

du(path, total=True, maxdepth=None, **kwargs)

Space used by files within a path

Parameters

path: str

total: bool

whether to sum all the file sizes

maxdepth: int or None

maximum number of directory levels to descend, None for unlimited.

kwargs: passed to ``ls``

Returns

Dict of {fn: size} if total=False, or int otherwise, where numbers

refer to bytes used.

end_transaction()

Finish write transaction, non-context version

exists(path, **kwargs)

Is there a file at the given path

expand_path(path, recursive=False, maxdepth=None)

Turn one or more globs or directories into a list of all matching paths to files or directories.

find(path, maxdepth=None, withdirs=False, **kwargs)

List all files below path.

Like posix find command without conditions

Parameters

pathstr

maxdepth: int or None

If not None, the maximum number of levels to descend

withdirs: bool

Whether to include directory paths in the output. This is True when used by glob, but users usually only want files.

kwargs are passed to ``ls``.

static from_json(blob)

Recreate a filesystem instance from JSON representation

See .to_json() for the expected structure of the input

Parameters

blob: str

Returns

file system instance, not necessarily of this particular class.

get(rpath, lpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, **kwargs)

Copy file(s) to local.

Copies a specific file or tree of files (if recursive=True). If lpath ends with a “/”, it will be assumed to be a directory, and target files will go within. Can submit a list of paths, which may be glob-patterns and will be expanded.

Calls get_file for each source.

get_file(rpath, lpath, callback=<fsspec.callbacks.NoOpCallback object>, **kwargs)

Copy single remote file to local

get_mapper(root, check=False, create=False)

Create key/value store based on this file-system

Makes a MutableMapping interface to the FS at the given root path. See fsspec.mapping.FSMap for further details.

glob(path, **kwargs)

Find files by glob-matching.

If the path ends with ‘/’ and does not contain “*”, it is essentially the same as ls(path), returning only files.

We support "**", "?" and "[..]". We do not support ^ for pattern negation.

Search path names that contain embedded characters special to this implementation of glob may not produce expected results; e.g., ‘foo/bar/starredfilename’.

kwargs are passed to ls.

head(path, size=1024)

Get the first size bytes from file

info(path, **kwargs)

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns

dict with keys: name (full path in the FS), size (in bytes), type (file,

directory, or something else) and other FS-specific keys.

invalidate_cache(path=None)

Discard any cached directory information

Parameters

path: string or None

If None, clear all listings cached else listings at or under given path.

isdir(path)

Is this entry directory-like?

isfile(path)

Is this entry file-like?

lexists(path, **kwargs)

If there is a file at the given path (including broken links)

listdir(path, detail=True, **kwargs)

Alias of AbstractFileSystem.ls.

ls(path, detail=True, **kwargs)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters

path: str

detail: bool

if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

kwargs: may have additional backend-specific options, such as version

information

Returns

List of strings if detail is False, or list of directory information

dicts if detail is True.

makedir(path, create_parents=True, **kwargs)

Alias of AbstractFileSystem.mkdir.

makedirs(path, exist_ok=False)

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

Parameters

path: str

leaf directory name

exist_ok: bool (False)

If False, will error if the target already exists

mkdir(path, create_parents=True, **kwargs)

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

Parameters

path: str

location

create_parents: bool

if True, this is equivalent to makedirs

kwargs:

may be permissions, etc.

mkdirs(path, exist_ok=False)

Alias of AbstractFileSystem.makedirs.

modified(path)

Return the modified timestamp of a file as a datetime.datetime

move(path1, path2, **kwargs)

Alias of AbstractFileSystem.mv.

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)

Move file(s) from one location to another

open(path, mode='rb', block_size=None, cache_options=None, compression=None, **kwargs)

Return a file-like object from the filesystem

The resultant instance must function correctly in a context with block.

Parameters

path: str

Target file

mode: str like ‘rb’, ‘w’

See builtin open()

block_size: int

Some indication of buffering - this is a value in bytes

cache_optionsdict, optional

Extra arguments to pass through to the cache.

compression: string or None

If given, open file using compression codec. Can either be a compression name (a key in fsspec.compression.compr) or “infer” to guess the compression from the filename suffix.

encoding, errors, newline: passed on to TextIOWrapper for text mode

pipe(path, value=None, **kwargs)

Put value into path

(counterpart to cat)

Parameters

path: string or dict(str, bytes)

If a string, a single remote location to put value bytes; if a dict, a mapping of {path: bytesvalue}.

value: bytes, optional

If using a single path, these are the bytes to put there. Ignored if path is a dict

pipe_file(path, value, **kwargs)

Set the bytes of given file

put(lpath, rpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, **kwargs)

Copy file(s) from local.

Copies a specific file or tree of files (if recursive=True). If rpath ends with a “/”, it will be assumed to be a directory, and target files will go within.

Calls put_file for each source.

put_file(lpath, rpath, callback=<fsspec.callbacks.NoOpCallback object>, **kwargs)

Copy single file to remote

read_block(fn, offset, length, delimiter=None)

Read a block of bytes from

Starting at offset of the file, read length bytes. If delimiter is set then we ensure that the read starts and stops at delimiter boundaries that follow the locations offset and offset + length. If offset is zero then we start at zero. The bytestring returned WILL include the end delimiter string.

If offset+length is beyond the eof, reads to eof.

Parameters

fn: string

Path to filename

offset: int

Byte offset to start read

length: int

Number of bytes to read

delimiter: bytes (optional)

Ensure reading starts and stops at delimiter bytestring

See also

utils.read_block

Examples

>>> fs.read_block('data/file.csv', 0, 13)  
b'Alice, 100\nBo'
>>> fs.read_block('data/file.csv', 0, 13, delimiter=b'\n')  
b'Alice, 100\nBob, 200\n'

Use length=None to read to the end of the file. >>> fs.read_block(‘data/file.csv’, 0, None, delimiter=b’n’) # doctest: +SKIP b’Alice, 100nBob, 200nCharlie, 300’

rename(path1, path2, **kwargs)

Alias of AbstractFileSystem.mv.

rm(path, recursive=False, maxdepth=None)

Delete files.

Parameters

path: str or list of str

File(s) to delete.

recursive: bool

If file(s) are directories, recursively delete contents and then also remove the directory

maxdepth: int or None

Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)

Delete a file

rmdir(path)

Remove a directory, if empty

sign(path, expiration=100, **kwargs)

Create a signed URL representing the given path

Some implementations allow temporary URLs to be generated, as a way of delegating credentials.

Parameters

pathstr

The path on the filesystem

expirationint

Number of seconds to enable the URL for (if supported)

Returns

URLstr

The signed URL

Raises

NotImplementedErrorif method is not implemented for a filesystem

size(path)

Size in bytes of file

sizes(paths)

Size in bytes of each file in a list of paths

start_transaction()

Begin write transaction for deferring files, non-context version

stat(path, **kwargs)

Alias of AbstractFileSystem.info.

tail(path, size=1024)

Get the last size bytes from file

to_json()

JSON representation of this filesystem instance

Returns

str: JSON structure with keys cls (the python location of this class),

protocol (text name of this class’s protocol, first one in case of multiple), args (positional args, usually empty), and all other kwargs as their own keys.

touch(path, truncate=True, **kwargs)

Create empty file, or update timestamp

Parameters

path: str

file location

truncate: bool

If True, always set file size to 0; if False, update timestamp and leave file unchanged, if backend allows this

property transaction

A context within which files are committed together upon exit

Requires the file class to implement commit() and discard() for the normal and exception cases.

ukey(path)

Hash of file properties, to tell if it has changed

upload(lpath, rpath, recursive=False, **kwargs)

Alias of AbstractFileSystem.put.

walk(path, maxdepth=None, **kwargs)

Return all files belows path

List all files, recursing into subdirectories; output is iterator-style, like os.walk(). For a simple list of files, find() is available.

Note that the “files” outputted will include anything that is not a directory, such as links.

Parameters

path: str

Root to recurse into

maxdepth: int

Maximum recursion depth. None means limitless, but not recommended on link-based file-systems.

kwargs: passed to ``ls``

class fsspec.spec.Transaction(fs)

Filesystem transaction write context

Gathers files for deferred commit or discard, so that several write operations can be finalized semi-atomically. This works by having this instance as the .transaction attribute of the given filesystem

Methods

complete([commit])	Finish transaction: commit or discard all deferred files
start()	Start a transaction on this FileSystem

complete(commit=True)

Finish transaction: commit or discard all deferred files

start()

Start a transaction on this FileSystem

class fsspec.spec.AbstractBufferedFile(fs, path, mode='rb', block_size='default', autocommit=True, cache_type='readahead', cache_options=None, size=None, **kwargs)

Convenient class to derive from to provide buffering

In the case that the backend does not provide a pythonic file-like object already, this class contains much of the logic to build one. The only methods that need to be overridden are _upload_chunk, _initiate_upload and _fetch_range.

Attributes

closed

details

full_name

Methods

close()	Close file
commit()	Move from temp to final destination
discard()	Throw away temporary file
fileno(/)	Returns underlying file descriptor if one exists.
flush([force])	Write buffered data to backend store.
info()	File information about this path
isatty(/)	Return whether this is an 'interactive' stream.
read([length])	Return data from cache, or fetch pieces as necessary
readable()	Whether opened for reading
readinto(b)	mirrors builtin file's readinto method
readline()	Read until first occurrence of newline character
readlines()	Return all data, split by the newline character
readuntil([char, blocks])	Return data between current position and first occurrence of char
seek(loc[, whence])	Set current file location
seekable()	Whether is seekable (only in read mode)
tell()	Current file location
truncate	Truncate file to size bytes.
writable()	Whether opened for writing
write(data)	Write data to buffer.
writelines(lines, /)	Write a list of lines to stream.

readinto1

close()

Close file

Finalizes writes, discards cache

commit()

Move from temp to final destination

discard()

Throw away temporary file

flush(force=False)

Write buffered data to backend store.

Writes the current buffer, if it is larger than the block-size, or if the file is being closed.

Parameters

force: bool

When closing, write the last block even if it is smaller than blocks are allowed to be. Disallows further writing to this file.

info()

File information about this path

read(length=- 1)

Return data from cache, or fetch pieces as necessary

Parameters

length: int (-1)

Number of bytes to read; if <0, all remaining bytes.

readable()

Whether opened for reading

readinto(b)

mirrors builtin file’s readinto method

https://docs.python.org/3/library/io.html#io.RawIOBase.readinto

readline()

Read until first occurrence of newline character

Note that, because of character encoding, this is not necessarily a true line ending.

readlines()

Return all data, split by the newline character

readuntil(char=b'\n', blocks=None)

Return data between current position and first occurrence of char

char is included in the output, except if the end of the tile is encountered first.

Parameters

char: bytes

Thing to find

blocks: None or int

How much to read in each go. Defaults to file blocksize - which may mean a new read on every call.

seek(loc, whence=0)

Set current file location

Parameters

loc: int

byte location

whence: {0, 1, 2}

from start of file, current location or end of file, resp.

seekable()

Whether is seekable (only in read mode)

tell()

Current file location

writable()

Whether opened for writing

write(data)

Write data to buffer.

Buffer only sent on flush() or if buffer is greater than or equal to blocksize.

Parameters

data: bytes

Set of bytes to be written.

class fsspec.archive.AbstractArchiveFileSystem(*args, **kwargs)

A generic superclass for implementing Archive-based filesystems.

Currently, it is shared amongst ZipFileSystem, LibArchiveFileSystem and TarFileSystem.

Attributes

transaction

A context within which files are committed together upon exit

Methods

cat(path[, recursive, on_error])	Fetch (potentially multiple) paths' contents
cat_file(path[, start, end])	Get the content of a file
checksum(path)	Unique value for current version of file
clear_instance_cache()	Clear the cache of filesystem instances.
copy(path1, path2[, recursive, on_error])	Copy within two locations in the filesystem
cp(path1, path2, **kwargs)	Alias of AbstractFileSystem.copy.
created(path)	Return the created timestamp of a file as a datetime.datetime
current()	Return the most recently created FileSystem
delete(path[, recursive, maxdepth])	Alias of AbstractFileSystem.rm.
disk_usage(path[, total, maxdepth])	Alias of AbstractFileSystem.du.
download(rpath, lpath[, recursive])	Alias of AbstractFileSystem.get.
du(path[, total, maxdepth])	Space used by files within a path
end_transaction()	Finish write transaction, non-context version
exists(path, **kwargs)	Is there a file at the given path
expand_path(path[, recursive, maxdepth])	Turn one or more globs or directories into a list of all matching paths to files or directories.
find(path[, maxdepth, withdirs])	List all files below path.
from_json(blob)	Recreate a filesystem instance from JSON representation
get(rpath, lpath[, recursive, callback])	Copy file(s) to local.
get_file(rpath, lpath[, callback])	Copy single remote file to local
get_mapper(root[, check, create])	Create key/value store based on this file-system
glob(path, **kwargs)	Find files by glob-matching.
head(path[, size])	Get the first size bytes from file
info(path, **kwargs)	Give details of entry at path
invalidate_cache([path])	Discard any cached directory information
isdir(path)	Is this entry directory-like?
isfile(path)	Is this entry file-like?
lexists(path, **kwargs)	If there is a file at the given path (including broken links)
listdir(path[, detail])	Alias of AbstractFileSystem.ls.
ls(path[, detail])	List objects at path.
makedir(path[, create_parents])	Alias of AbstractFileSystem.mkdir.
makedirs(path[, exist_ok])	Recursively make directories
mkdir(path[, create_parents])	Create directory entry at path
mkdirs(path[, exist_ok])	Alias of AbstractFileSystem.makedirs.
modified(path)	Return the modified timestamp of a file as a datetime.datetime
move(path1, path2, **kwargs)	Alias of AbstractFileSystem.mv.
mv(path1, path2[, recursive, maxdepth])	Move file(s) from one location to another
open(path[, mode, block_size, ...])	Return a file-like object from the filesystem
pipe(path[, value])	Put value into path
pipe_file(path, value, **kwargs)	Set the bytes of given file
put(lpath, rpath[, recursive, callback])	Copy file(s) from local.
put_file(lpath, rpath[, callback])	Copy single file to remote
read_block(fn, offset, length[, delimiter])	Read a block of bytes from
rename(path1, path2, **kwargs)	Alias of AbstractFileSystem.mv.
rm(path[, recursive, maxdepth])	Delete files.
rm_file(path)	Delete a file
rmdir(path)	Remove a directory, if empty
sign(path[, expiration])	Create a signed URL representing the given path
size(path)	Size in bytes of file
sizes(paths)	Size in bytes of each file in a list of paths
start_transaction()	Begin write transaction for deferring files, non-context version
stat(path, **kwargs)	Alias of AbstractFileSystem.info.
tail(path[, size])	Get the last size bytes from file
to_json()	JSON representation of this filesystem instance
touch(path[, truncate])	Create empty file, or update timestamp
ukey(path)	Hash of file properties, to tell if it has changed
upload(lpath, rpath[, recursive])	Alias of AbstractFileSystem.put.
walk(path[, maxdepth])	Return all files belows path

cat_ranges
cp_file

info(path, **kwargs)

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns

dict with keys: name (full path in the FS), size (in bytes), type (file,

directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters

path: str

detail: bool

if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

kwargs: may have additional backend-specific options, such as version

information

Returns

List of strings if detail is False, or list of directory information

dicts if detail is True.

ukey(path)

Hash of file properties, to tell if it has changed

class fsspec.FSMap(root, fs, check=False, create=False, missing_exceptions=None)

Wrap a FileSystem instance as a mutable wrapping.

The keys of the mapping become files under the given root, and the values (which must be bytes) the contents of those files.

Parameters

root: string

prefix for all the files

fs: FileSystem instance

check: bool (=True)

performs a touch at the location, to check for write access.

Examples

>>> fs = FileSystem(**parameters)  
>>> d = FSMap('my-data/path/', fs)  
or, more likely
>>> d = fs.get_mapper('my-data/path/')

>>> d['loc1'] = b'Hello World'  
>>> list(d.keys())  
['loc1']
>>> d['loc1']  
b'Hello World'

Methods

clear()	Remove all keys below root - empties out mapping
delitems(keys)	Remove multiple keys from the store
get(k[,d])
getitems(keys[, on_error])	Fetch multiple items from the store
items()
keys()
pop(k[,d])	If key is not found, d is returned if given, otherwise KeyError is raised.
popitem()	as a 2-tuple; but raise KeyError if D is empty.
setdefault(k[,d])
setitems(values_dict)	Set the values of multiple items in the store
update([E, ]**F)	If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
values()

clear()

Remove all keys below root - empties out mapping

delitems(keys)

Remove multiple keys from the store

getitems(keys, on_error='raise')

Fetch multiple items from the store

If the backend is async-able, this might proceed concurrently

Parameters

keys: list(str)

They keys to be fetched

on_error“raise”, “omit”, “return”

If raise, an underlying exception will be raised (converted to KeyError if the type is in self.missing_exceptions); if omit, keys with exception will simply not be included in the output; if “return”, all keys are included in the output, but the value will be bytes or an exception instance.

Returns

dict(key, bytes|exception)

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

setitems(values_dict)

Set the values of multiple items in the store

Parameters

values_dict: dict(str, bytes)

class fsspec.core.OpenFile(fs, path, mode='rb', compression=None, encoding=None, errors=None, newline=None)

File-like object to be used in a context

Can layer (buffered) text-mode and compression over any file-system, which are typically binary-only.

These instances are safe to serialize, as the low-level file object is not created until invoked using with.

Parameters

fs: FileSystem

The file system to use for opening the file. Should be a subclass or duck-type with fsspec.spec.AbstractFileSystem

path: str

Location to open

mode: str like ‘rb’, optional

Mode of the opened file

compression: str or None, optional

Compression to apply

encoding: str or None, optional

The encoding to use if opened in text mode.

errors: str or None, optional

How to handle encoding errors if opened in text mode.

newline: None or str

Passed to TextIOWrapper in text mode, how to handle line endings.

Attributes

full_name

Methods

close()	Close all encapsulated file objects
open()	Materialise this as a real open file without context

close()

Close all encapsulated file objects

open()

Materialise this as a real open file without context

The file should be explicitly closed to avoid enclosed file instances persisting. This code-path monkey-patches the file-like objects, so they can close even if the parent OpenFile object has already been deleted; but a with-context is better style.

class fsspec.core.OpenFiles(*args, mode='rb', fs=None)

List of OpenFile instances

Can be used in a single context, which opens and closes all of the contained files. Normal list access to get the elements works as normal.

A special case is made for caching filesystems - the files will be down/uploaded together at the start or end of the context, and this may happen concurrently, if the target filesystem supports it.

Methods

append(object, /)	Append object to the end of the list.
clear(/)	Remove all items from list.
copy(/)	Return a shallow copy of the list.
count(value, /)	Return number of occurrences of value.
extend(iterable, /)	Extend list by appending elements from the iterable.
index(value[, start, stop])	Return first index of value.
insert(index, object, /)	Insert object before index.
pop([index])	Remove and return item at index (default last).
remove(value, /)	Remove first occurrence of value.
reverse(/)	Reverse IN PLACE.
sort(*[, key, reverse])	Sort the list in ascending order and return None.

class fsspec.core.BaseCache(blocksize, fetcher, size)

Pass-though cache: doesn’t keep anything, calls every time

Acts as base class for other cachers

Parameters

blocksize: int

How far to read ahead in numbers of bytes

fetcher: func

Function of the form f(start, end) which gets bytes from remote as specified

size: int

How big this file is

fsspec.core.get_fs_token_paths(urlpath, mode='rb', num=1, name_function=None, storage_options=None, protocol=None, expand=True)

Filesystem, deterministic token, and paths from a urlpath and options.

Parameters

urlpath: string or iterable

Absolute or relative filepath, URL (may include protocols like s3://), or globstring pointing to data.

mode: str, optional

Mode in which to open files.

num: int, optional

If opening in writing mode, number of files we expect to create.

name_function: callable, optional

If opening in writing mode, this callable is used to generate path names. Names are generated for each partition by urlpath.replace('*', name_function(partition_index)).

storage_options: dict, optional

Additional keywords to pass to the filesystem class.

protocol: str or None

To override the protocol specifier in the URL

expand: bool

Expand string paths for writing, assuming the path is a directory

fsspec.core.url_to_fs(url, **kwargs)

Turn fully-qualified and potentially chained URL into filesystem instance

Parameters

urlstr

The fsspec-compatible URL

**kwargs: dict

Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc.

Returns

filesystemFileSystem

The new filesystem discovered from url and created with **kwargs.

urlpathstr

The file-systems-specific URL for url.

class fsspec.dircache.DirCache(use_listings_cache=True, listings_expiry_time=None, max_paths=None, **kwargs)

Caching of directory listings, in a structure like:

{"path0": [
    {"name": "path0/file0",
     "size": 123,
     "type": "file",
     ...
    },
    {"name": "path0/file1",
    },
    ...
    ],
 "path1": [...]
}

Parameters to this class control listing expiry or indeed turn caching off

__init__(use_listings_cache=True, listings_expiry_time=None, max_paths=None, **kwargs)

Parameters

use_listings_cache: bool

If False, this cache never returns items, but always reports KeyError, and setting items has no effect

listings_expiry_time: int or float (optional)

Time in seconds that a listing is considered valid. If None, listings do not expire.

max_paths: int (optional)

The number of most recent listings that are considered valid; ‘recent’ refers to when the entry was set.

class fsspec.registry.ReadOnlyRegistry(target)

Dict-like registry, but immutable

Maps backend name to implementation class

To add backend implementations, use register_implementation

__init__(target)

fsspec.registry.register_implementation(name, cls, clobber=True, errtxt=None)

Add implementation class to the registry

Parameters

name: str

Protocol name to associate with the class

cls: class or str

if a class: fsspec-compliant implementation class (normally inherits from fsspec.AbstractFileSystem, gets added straight to the registry. If a str, the full path to an implementation class like package.module.class, which gets added to known_implementations, so the import is deferred until the filesystem is actually used.

clobber: bool (optional)

Whether to overwrite a protocol with the same name; if False, will raise instead.

errtxt: str (optional)

If given, then a failure to import the given class will result in this text being given.

class fsspec.callbacks.Callback(size=None, value=0, hooks=None, **kwargs)

Base class and interface for callback mechanism

This class can be used directly for monitoring file transfers by providing callback=Callback(hooks=...) (see the hooks argument, below), or subclassed for more specialised behaviour.

Parameters

size: int (optional)

Nominal quantity for the value that corresponds to a complete transfer, e.g., total number of tiles or total number of bytes

value: int (0)

Starting internal counter value

hooks: dict or None

A dict of named functions to be called on each update. The signature of these must be f(size, value, **kwargs)

Methods

absolute_update(value)	Set the internal value state
as_callback([maybe_callback])	Transform callback=.
branch(path_1, path_2, kwargs)	Set callbacks for child transfers
call([hook_name])	Execute hook(s) with current state
relative_update([inc])	Delta increment the internal cuonter
set_size(size)	Set the internal maximum size attribute
wrap(iterable)	Wrap an iterable to call relative_update on each iterations

no_op

absolute_update(value)

Set the internal value state

Triggers call()

Parameters

value: int

classmethod as_callback(maybe_callback=None)

Transform callback=… into Callback instance

For the special value of None, return the global instance of NoOpCallback. This is an alternative to including callback=_DEFAULT_CALLBACK directly in a method signature.

branch(path_1, path_2, kwargs)

Set callbacks for child transfers

If this callback is operating at a higher level, e.g., put, which may trigger transfers that can also be monitored. The passed kwargs are to be mutated to add callback=, if this class supports branching to children.

Parameters

path_1: str

Child’s source path

path_2: str

Child’s destination path

kwargs: dict

arguments passed to child method, e.g., put_file.

call(hook_name=None, **kwargs)

Execute hook(s) with current state

Each function is passed the internal size and current value

Parameters

hook_name: str or None

If given, execute on this hook

kwargs: passed on to (all) hook(s)

relative_update(inc=1)

Delta increment the internal cuonter

Triggers call()

Parameters

inc: int

set_size(size)

Set the internal maximum size attribute

Usually called if not initially set at instantiation. Note that this triggers a call().

Parameters

size: int

wrap(iterable)

Wrap an iterable to call relative_update on each iterations

Parameters

iterable: Iterable

The iterable that is being wrapped

class fsspec.callbacks.NoOpCallback(size=None, value=0, hooks=None, **kwargs)

This implementation of Callback does exactly nothing

Methods

absolute_update(value)	Set the internal value state
as_callback([maybe_callback])	Transform callback=.
branch(path_1, path_2, kwargs)	Set callbacks for child transfers
call(*args, **kwargs)	Execute hook(s) with current state
relative_update([inc])	Delta increment the internal cuonter
set_size(size)	Set the internal maximum size attribute
wrap(iterable)	Wrap an iterable to call relative_update on each iterations

no_op

call(*args, **kwargs)

Execute hook(s) with current state

Each function is passed the internal size and current value

Parameters

hook_name: str or None

If given, execute on this hook

kwargs: passed on to (all) hook(s)

class fsspec.callbacks.DotPrinterCallback(chr_to_print='#', **kwargs)

Simple example Callback implementation

Almost identical to Callback with a hook that prints a char; here we demonstrate how the outer layer may print “#” and the inner layer “.”

Methods

absolute_update(value)	Set the internal value state
as_callback([maybe_callback])	Transform callback=.
branch(path_1, path_2, kwargs)	Mutate kwargs to add new instance with different print char
call(**kwargs)	Just outputs a character
relative_update([inc])	Delta increment the internal cuonter
set_size(size)	Set the internal maximum size attribute
wrap(iterable)	Wrap an iterable to call relative_update on each iterations

no_op

branch(path_1, path_2, kwargs)

Mutate kwargs to add new instance with different print char

call(**kwargs)

Just outputs a character

Built-in Implementations

fsspec.implementations.ftp.FTPFileSystem(...)	A filesystem over classic FTP
fsspec.implementations.hdfs.PyArrowHDFS(...)	Adapted version of Arrow's HadoopFileSystem
fsspec.implementations.arrow.ArrowFSWrapper(...)	FSSpec-compatible wrapper of pyarrow.fs.FileSystem.
fsspec.implementations.arrow.HadoopFileSystem(...)	A wrapper on top of the pyarrow.fs.HadoopFileSystem to connect it's interface with fsspec
fsspec.implementations.dask.DaskWorkerFileSystem(...)	View files accessible to a worker as any other remote file-system
fsspec.implementations.http.HTTPFileSystem(...)	Simple File-System for fetching data via HTTP(S)
fsspec.implementations.local.LocalFileSystem(...)	Interface to files on local storage
fsspec.implementations.memory.MemoryFileSystem(...)	A filesystem based on a dict of BytesIO objects
fsspec.implementations.github.GithubFileSystem(...)	Interface to files in github
fsspec.implementations.sftp.SFTPFileSystem(...)	Files over SFTP/SSH
fsspec.implementations.webhdfs.WebHDFS(...)	Interface to HDFS over HTTP using the WebHDFS API.
fsspec.implementations.zip.ZipFileSystem(...)	Read contents of ZIP archive as a file-system
fsspec.implementations.cached.CachingFileSystem(...)	Locally caching filesystem, layer over any other FS
fsspec.implementations.cached.WholeFileCacheFileSystem(...)	Caches whole remote files on first access
fsspec.implementations.cached.SimpleCacheFileSystem(...)	Caches whole remote files on first access
fsspec.implementations.git.GitFileSystem(...)	Browse the files of a local git repo at any hash/tag/branch
fsspec.implementations.jupyter.JupyterFileSystem(...)	View of the files as seen by a Jupyter server (notebook or lab)
fsspec.implementations.libarchive.LibArchiveFileSystem(...)	Compressed archives as a file-system (read-only)
fsspec.implementations.dbfs.DatabricksFileSystem(...)	Get access to the Databricks filesystem implementation over HTTP.
fsspec.implementations.reference.ReferenceFileSystem(...)	View byte ranges of some other file as a file system

class fsspec.implementations.ftp.FTPFileSystem(*args, **kwargs)

A filesystem over classic FTP

__init__(host, port=21, username=None, password=None, acct=None, block_size=None, tempdir=None, timeout=30, **kwargs)

You can use _get_kwargs_from_urls to get some kwargs from a reasonable FTP url.

Authentication will be anonymous if username/password are not given.

Parameters

host: str

The remote server name/ip to connect to

port: int

Port to connect with

username: str or None

If authenticating, the user’s identifier

password: str of None

User’s password on the server, if using

acct: str or None

Some servers also need an “account” string for auth

block_size: int or None

If given, the read-ahead or write buffer size.

tempdir: str

Directory on remote to put temporary files when in a transaction

timeout: int

Timeout of the ftp connection in seconds

class fsspec.implementations.hdfs.PyArrowHDFS(*args, **kwargs)

Adapted version of Arrow’s HadoopFileSystem

This is a very simple wrapper over the pyarrow.hdfs.HadoopFileSystem, which passes on all calls to the underlying class.

__init__(host='default', port=0, user=None, kerb_ticket=None, driver='libhdfs', extra_conf=None, **kwargs)

Parameters

host: str

Hostname, IP or “default” to try to read from Hadoop config

port: int

Port to connect on, or default from Hadoop config if 0

user: str or None

If given, connect as this username

kerb_ticket: str or None

If given, use this ticket for authentication

driver: ‘libhdfs’ or ‘libhdfs3’

Binary driver; libhdfs if the JNI library and default

extra_conf: None or dict

Passed on to HadoopFileSystem

class fsspec.implementations.arrow.ArrowFSWrapper(*args, **kwargs)

FSSpec-compatible wrapper of pyarrow.fs.FileSystem.

Parameters

fspyarrow.fs.FileSystem

__init__(fs, **kwargs)

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters

use_listings_cache, listings_expiry_time, max_paths:

passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

skip_instance_cache: bool

If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

asynchronous: bool

loop: asyncio-compatible IOLoop or None

class fsspec.implementations.dask.DaskWorkerFileSystem(*args, **kwargs)

View files accessible to a worker as any other remote file-system

When instances are run on the worker, uses the real filesystem. When run on the client, they call the worker to provide information or data.

Warning this implementation is experimental, and read-only for now.

__init__(target_protocol=None, target_options=None, fs=None, client=None, **kwargs)

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters

use_listings_cache, listings_expiry_time, max_paths:

passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

skip_instance_cache: bool

If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

asynchronous: bool

loop: asyncio-compatible IOLoop or None

class fsspec.implementations.http.HTTPFileSystem(*args, **kwargs)

Simple File-System for fetching data via HTTP(S)

ls() is implemented by loading the parent page and doing a regex match on the result. If simple_link=True, anything of the form “http(s)://server.com/stuff?thing=other”; otherwise only links within HTML href tags will be used.

__init__(simple_links=True, block_size=None, same_scheme=True, size_policy=None, cache_type='bytes', cache_options=None, asynchronous=False, loop=None, client_kwargs=None, get_client=<function get_client>, **storage_options)

NB: if this is called async, you must await set_client

Parameters

block_size: int

Blocks to read bytes; if 0, will default to raw requests file-like objects instead of HTTPFile instances

simple_links: bool

If True, will consider both HTML <a> tags and anything that looks like a URL; if False, will consider only the former.

same_scheme: True

When doing ls/glob, if this is True, only consider paths that have http/https matching the input URLs.

size_policy: this argument is deprecated

client_kwargs: dict

Passed to aiohttp.ClientSession, see https://docs.aiohttp.org/en/stable/client_reference.html For example, {'auth': aiohttp.BasicAuth('user', 'pass')}

get_client: Callable[…, aiohttp.ClientSession]

A callable which takes keyword arguments and constructs an aiohttp.ClientSession. It’s state will be managed by the HTTPFileSystem class.

storage_options: key-value

Any other parameters passed on to requests

cache_type, cache_options: defaults used in open

class fsspec.implementations.local.LocalFileSystem(*args, **kwargs)

Interface to files on local storage

Parameters

auto_mkdir: bool

Whether, when opening a file, the directory containing it should be created (if it doesn’t already exist). This is assumed by pyarrow code.

__init__(auto_mkdir=False, **kwargs)

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters

use_listings_cache, listings_expiry_time, max_paths:

passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

skip_instance_cache: bool

If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

asynchronous: bool

loop: asyncio-compatible IOLoop or None

class fsspec.implementations.memory.MemoryFileSystem(*args, **kwargs)

A filesystem based on a dict of BytesIO objects

This is a global filesystem so instances of this class all point to the same in memory filesystem.

__init__(*args, **storage_options)

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters

use_listings_cache, listings_expiry_time, max_paths:

passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

skip_instance_cache: bool

If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

asynchronous: bool

loop: asyncio-compatible IOLoop or None

class fsspec.implementations.sftp.SFTPFileSystem(*args, **kwargs)

Files over SFTP/SSH

Peer-to-peer filesystem over SSH using paramiko.

Note: if using this with the open or open_files, with full URLs, there is no way to tell if a path is relative, so all paths are assumed to be absolute.

__init__(host, **ssh_kwargs)

Parameters

host: str

Hostname or IP as a string

temppath: str

Location on the server to put files, when within a transaction

ssh_kwargs: dict

Parameters passed on to connection. See details in http://docs.paramiko.org/en/2.4/api/client.html#paramiko.client.SSHClient.connect May include port, username, password…

class fsspec.implementations.webhdfs.WebHDFS(*args, **kwargs)

Interface to HDFS over HTTP using the WebHDFS API. Supports also HttpFS gateways.

Three auth mechanisms are supported:

insecure: no auth is done, and the user is assumed to be whoever they

say they are (parameter user), or a predefined value such as “dr.who” if not given

spnego: when kerberos authentication is enabled, auth is negotiated by

requests_kerberos https://github.com/requests/requests-kerberos . This establishes a session based on existing kinit login and/or specified principal/password; parameters are passed with kerb_kwargs

token: uses an existing Hadoop delegation token from another secured

service. Indeed, this client can also generate such tokens when not insecure. Note that tokens expire, but can be renewed (by a previously specified user) and may allow for proxying.

__init__(host, port=50070, kerberos=False, token=None, user=None, proxy_to=None, kerb_kwargs=None, data_proxy=None, use_https=False, **kwargs)

Parameters

host: str

Name-node address

port: int

Port for webHDFS

kerberos: bool

Whether to authenticate with kerberos for this connection

token: str or None

If given, use this token on every call to authenticate. A user and user-proxy may be encoded in the token and should not be also given

user: str or None

If given, assert the user name to connect with

proxy_to: str or None

If given, the user has the authority to proxy, and this value is the user in who’s name actions are taken

kerb_kwargs: dict

Any extra arguments for HTTPKerberosAuth, see https://github.com/requests/requests-kerberos/blob/master/requests_kerberos/kerberos_.py

data_proxy: dict, callable or None

If given, map data-node addresses. This can be necessary if the HDFS cluster is behind a proxy, running on Docker or otherwise has a mismatch between the host-names given by the name-node and the address by which to refer to them from the client. If a dict, maps host names host->data_proxy[host]; if a callable, full URLs are passed, and function must conform to url->data_proxy(url).

use_https: bool

Whether to connect to the Name-node using HTTPS instead of HTTP

kwargs

class fsspec.implementations.zip.ZipFileSystem(*args, **kwargs)

Read contents of ZIP archive as a file-system

Keeps file object open while instance lives.

This class is pickleable, but not necessarily thread-safe

__init__(fo='', mode='r', target_protocol=None, target_options=None, block_size=5242880, **kwargs)

Parameters

fo: str or file-like

Contains ZIP, and must exist. If a str, will fetch file using open_files(), which must return one file exactly.

mode: str

Currently, only ‘r’ accepted

target_protocol: str (optional)

If fo is a string, this value can be used to override the FS protocol inferred from a URL

target_options: dict (optional)

Kwargs passed when instantiating the target FS, if fo is a string.

class fsspec.implementations.cached.CachingFileSystem(*args, **kwargs)

Locally caching filesystem, layer over any other FS

This class implements chunk-wise local storage of remote files, for quick access after the initial download. The files are stored in a given directory with random hashes for the filenames. If no directory is given, a temporary one is used, which should be cleaned up by the OS after the process ends. The files themselves as sparse (as implemented in MMapCache), so only the data which is accessed takes up space.

Restrictions:

• the block-size must be the same for each access of a given file, unless all blocks of the file have already been read

• caching can only be applied to file-systems which produce files derived from fsspec.spec.AbstractBufferedFile ; LocalFileSystem is also allowed, for testing

__init__(target_protocol=None, cache_storage='TMP', cache_check=10, check_files=False, expiry_time=604800, target_options=None, fs=None, same_names=False, compression=None, **kwargs)

Parameters

target_protocol: str (optional)

Target filesystem protocol. Provide either this or fs.

cache_storage: str or list(str)

Location to store files. If “TMP”, this is a temporary directory, and will be cleaned up by the OS when this process ends (or later). If a list, each location will be tried in the order given, but only the last will be considered writable.

cache_check: int

Number of seconds between reload of cache metadata

check_files: bool

Whether to explicitly see if the UID of the remote file matches the stored one before using. Warning: some file systems such as HTTP cannot reliably give a unique hash of the contents of some path, so be sure to set this option to False.

expiry_time: int

The time in seconds after which a local copy is considered useless. Set to falsy to prevent expiry. The default is equivalent to one week.

target_options: dict or None

Passed to the instantiation of the FS, if fs is None.

fs: filesystem instance

The target filesystem to run against. Provide this or protocol.

same_names: bool (optional)

By default, target URLs are hashed, so that files from different backends with the same basename do not conflict. If this is true, the original basename is used.

compression: str (optional)

To decompress on download. Can be ‘infer’ (guess from the URL name), one of the entries in fsspec.compression.compr, or None for no decompression.

class fsspec.implementations.cached.WholeFileCacheFileSystem(*args, **kwargs)

Caches whole remote files on first access

This class is intended as a layer over any other file system, and will make a local copy of each file accessed, so that all subsequent reads are local. This is similar to CachingFileSystem, but without the block-wise functionality and so can work even when sparse files are not allowed. See its docstring for definition of the init arguments.

The class still needs access to the remote store for listing files, and may refresh cached files.

__init__(target_protocol=None, cache_storage='TMP', cache_check=10, check_files=False, expiry_time=604800, target_options=None, fs=None, same_names=False, compression=None, **kwargs)

Parameters

target_protocol: str (optional)

Target filesystem protocol. Provide either this or fs.

cache_storage: str or list(str)

Location to store files. If “TMP”, this is a temporary directory, and will be cleaned up by the OS when this process ends (or later). If a list, each location will be tried in the order given, but only the last will be considered writable.

cache_check: int

Number of seconds between reload of cache metadata

check_files: bool

Whether to explicitly see if the UID of the remote file matches the stored one before using. Warning: some file systems such as HTTP cannot reliably give a unique hash of the contents of some path, so be sure to set this option to False.

expiry_time: int

The time in seconds after which a local copy is considered useless. Set to falsy to prevent expiry. The default is equivalent to one week.

target_options: dict or None

Passed to the instantiation of the FS, if fs is None.

fs: filesystem instance

The target filesystem to run against. Provide this or protocol.

same_names: bool (optional)

By default, target URLs are hashed, so that files from different backends with the same basename do not conflict. If this is true, the original basename is used.

compression: str (optional)

To decompress on download. Can be ‘infer’ (guess from the URL name), one of the entries in fsspec.compression.compr, or None for no decompression.

class fsspec.implementations.cached.SimpleCacheFileSystem(*args, **kwargs)

Caches whole remote files on first access

This class is intended as a layer over any other file system, and will make a local copy of each file accessed, so that all subsequent reads are local. This implementation only copies whole files, and does not keep any metadata about the download time or file details. It is therefore safer to use in multi-threaded/concurrent situations.

This is the only of the caching filesystems that supports write: you will be given a real local open file, and upon close and commit, it will be uploaded to the target filesystem; the writability or the target URL is not checked until that time.

__init__(**kwargs)

Parameters

target_protocol: str (optional)

Target filesystem protocol. Provide either this or fs.

cache_storage: str or list(str)

Location to store files. If “TMP”, this is a temporary directory, and will be cleaned up by the OS when this process ends (or later). If a list, each location will be tried in the order given, but only the last will be considered writable.

cache_check: int

Number of seconds between reload of cache metadata

check_files: bool

Whether to explicitly see if the UID of the remote file matches the stored one before using. Warning: some file systems such as HTTP cannot reliably give a unique hash of the contents of some path, so be sure to set this option to False.

expiry_time: int

The time in seconds after which a local copy is considered useless. Set to falsy to prevent expiry. The default is equivalent to one week.

target_options: dict or None

Passed to the instantiation of the FS, if fs is None.

fs: filesystem instance

The target filesystem to run against. Provide this or protocol.

same_names: bool (optional)

By default, target URLs are hashed, so that files from different backends with the same basename do not conflict. If this is true, the original basename is used.

compression: str (optional)

To decompress on download. Can be ‘infer’ (guess from the URL name), one of the entries in fsspec.compression.compr, or None for no decompression.

class fsspec.implementations.github.GithubFileSystem(*args, **kwargs)

Interface to files in github

An instance of this class provides the files residing within a remote github repository. You may specify a point in the repos history, by SHA, branch or tag (default is current master).

Given that code files tend to be small, and that github does not support retrieving partial content, we always fetch whole files.

When using fsspec.open, allows URIs of the form:

• “github://path/file”, in which case you must specify org, repo and may specify sha in the extra args

• ‘github://org:repo@/precip/catalog.yml’, where the org and repo are part of the URI

• ‘github://org:repo@sha/precip/catalog.yml’, where the sha is also included

sha can be the full or abbreviated hex of the commit you want to fetch from, or a branch or tag name (so long as it doesn’t contain special characters like “/”, “?”, which would have to be HTTP-encoded).

For authorised access, you must provide username and token, which can be made at https://github.com/settings/tokens

__init__(org, repo, sha=None, username=None, token=None, **kwargs)

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters

use_listings_cache, listings_expiry_time, max_paths:

passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

skip_instance_cache: bool

If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

asynchronous: bool

loop: asyncio-compatible IOLoop or None

class fsspec.implementations.git.GitFileSystem(*args, **kwargs)

Browse the files of a local git repo at any hash/tag/branch

(experimental backend)

__init__(path=None, fo=None, ref=None, **kwargs)

Parameters

path: str (optional)

Local location of the repo (uses current directory if not given). May be deprecated in favour of fo. When used with a higher level function such as fsspec.open(), may be of the form “git://[path-to-repo[:]][ref@]path/to/file” (but the actual file path should not contain “@” or “:”).

fo: str (optional)

Same as path, but passed as part of a chained URL. This one takes precedence if both are given.

ref: str (optional)

Reference to work with, could be a hash, tag or branch name. Defaults to current working tree. Note that ls and open also take hash, so this becomes the default for those operations

kwargs

class fsspec.implementations.jupyter.JupyterFileSystem(*args, **kwargs)

View of the files as seen by a Jupyter server (notebook or lab)

__init__(url, tok=None, **kwargs)

Parameters

urlstr

Base URL of the server, like “http://127.0.0.1:8888”. May include token in the string, which is given by the process when starting up

tokstr

If the token is obtained separately, can be given here

kwargs

class fsspec.implementations.libarchive.LibArchiveFileSystem(*args, **kwargs)

Compressed archives as a file-system (read-only)

Supports the following formats: tar, pax , cpio, ISO9660, zip, mtree, shar, ar, raw, xar, lha/lzh, rar Microsoft CAB, 7-Zip, WARC

See the libarchive documentation for further restrictions. https://www.libarchive.org/

Keeps file object open while instance lives. It only works in seekable file-like objects. In case the filesystem does not support this kind of file object, it is recommended to cache locally.

This class is pickleable, but not necessarily thread-safe (depends on the platform). See libarchive documentation for details.

__init__(fo='', mode='r', target_protocol=None, target_options=None, block_size=5242880, **kwargs)

Parameters

fo: str or file-like

Contains ZIP, and must exist. If a str, will fetch file using open_files(), which must return one file exactly.

mode: str

Currently, only ‘r’ accepted

target_protocol: str (optional)

If fo is a string, this value can be used to override the FS protocol inferred from a URL

target_options: dict (optional)

Kwargs passed when instantiating the target FS, if fo is a string.

class fsspec.implementations.dbfs.DatabricksFileSystem(*args, **kwargs)

Get access to the Databricks filesystem implementation over HTTP. Can be used inside and outside of a databricks cluster.

__init__(instance, token, **kwargs)

Create a new DatabricksFileSystem.

Parameters

instance: str

The instance URL of the databricks cluster. For example for an Azure databricks cluster, this has the form adb-<some-number>.<two digits>.azuredatabricks.net.

token: str

Your personal token. Find out more here: https://docs.databricks.com/dev-tools/api/latest/authentication.html

class fsspec.implementations.reference.ReferenceFileSystem(*args, **kwargs)

View byte ranges of some other file as a file system

Initial version: single file system target, which must support async, and must allow start and end args in _cat_file. Later versions may allow multiple arbitrary URLs for the targets.

This FileSystem is read-only. It is designed to be used with async targets (for now). This FileSystem only allows whole-file access, no open. We do not get original file details from the target FS.

Configuration is by passing a dict of references at init, or a URL to a JSON file containing the same; this dict can also contain concrete data for some set of paths.

Reference dict format: {path0: bytes_data, path1: (target_url, offset, size)}

https://github.com/fsspec/kerchunk/blob/main/README.md

__init__(fo, target=None, ref_storage_args=None, target_protocol=None, target_options=None, remote_protocol=None, remote_options=None, fs=None, template_overrides=None, simple_templates=True, loop=None, ref_type=None, **kwargs)

Parameters

fodict or str

The set of references to use for this instance, with a structure as above. If str, will use fsspec.open, in conjunction with ref_storage_args to open and parse JSON at this location.

targetstr

For any references having target_url as None, this is the default file target to use

ref_storage_argsdict

If references is a str, use these kwargs for loading the JSON file

target_protocolstr

Used for loading the reference file, if it is a path. If None, protocol will be derived from the given path

target_optionsdict

Extra FS options for loading the reference file, if given as a path

remote_protocolstr

The protocol of the filesystem on which the references will be evaluated (unless fs is provided). If not given, will be derived from the first URL that has a protocol in the templates or in the references, in that order.

remote_optionsdict

kwargs to go with remote_protocol

fsfile system instance

Directly provide a file system, if you want to configure it beforehand. This takes precedence over target_protocol/target_options

template_overridesdict

Swap out any templates in the references file with these - useful for testing.

ref_type“json” | “parquet” | “zarr”

If None, guessed from URL suffix, defaulting to JSON. Ignored if fo is not a string.

simple_templates: bool

Whether templates can be processed with simple replace (True) or if jinja is needed (False, much slower). All reference sets produced by kerchunk are simple in this sense, but the spec allows for complex.

kwargspassed to parent class

Other Known Implementations

• s3fs for Amazon S3 and other compatible stores

• gcsfs for Google Cloud Storage

• adl for Azure DataLake storage

• abfs for Azure Blob service

• dropbox for access to dropbox shares

• ocifs for access to Oracle Cloud Object Storage

• gdrive to access Google Drive and shares (experimental)

• wandbfs to access Wandb run data (experimental)

Read Buffering

fsspec.caching.ReadAheadCache(blocksize, ...)	Cache which reads only when we get beyond a block of data
fsspec.caching.BytesCache(blocksize, ...[, trim])	Cache which holds data in a in-memory bytes object
fsspec.caching.MMapCache(blocksize, fetcher, ...)	memory-mapped sparse file cache
fsspec.caching.BlockCache(blocksize, ...[, ...])	Cache holding memory as a set of blocks.

class fsspec.caching.ReadAheadCache(blocksize, fetcher, size)

Cache which reads only when we get beyond a block of data

This is a much simpler version of BytesCache, and does not attempt to fill holes in the cache or keep fragments alive. It is best suited to many small reads in a sequential order (e.g., reading lines from a file).

class fsspec.caching.BytesCache(blocksize, fetcher, size, trim=True)

Cache which holds data in a in-memory bytes object

Implements read-ahead by the block size, for semi-random reads progressing through the file.

Parameters

trim: bool

As we read more data, whether to discard the start of the buffer when we are more than a blocksize ahead of it.

class fsspec.caching.MMapCache(blocksize, fetcher, size, location=None, blocks=None)

memory-mapped sparse file cache

Opens temporary file, which is filled blocks-wise when data is requested. Ensure there is enough disc space in the temporary location.

This cache method might only work on posix

class fsspec.caching.BlockCache(blocksize, fetcher, size, maxblocks=32)

Cache holding memory as a set of blocks.

Requests are only ever made blocksize at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than maxblocks are stored.

Parameters

blocksizeint

The number of bytes to store in each block. Requests are only ever made for blocksize, so this should balance the overhead of making a request against the granularity of the blocks.

fetcherCallable

sizeint

The total size of the file being cached.

maxblocksint

The maximum number of blocks to cache for. The maximum memory use for this cache is then blocksize * maxblocks.

Methods

cache_info()

The statistics on the block cache.

cache_info()

The statistics on the block cache.

Returns

NamedTuple

Returned directly from the LRU Cache used internally.