pyspark.sql.DataFrameReader.options#

DataFrameReader.options(**options)[source]#

Adds input options for the underlying data source.

New in version 1.4.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

**optionsdict: The dictionary of string keys and prmitive-type values.

Examples

>>> spark.read.options(key="value")
<...readwriter.DataFrameReader object ...>

Specify options in a dictionary.

>>> spark.read.options(**{"k1": "v1", "k2": "v2"})
<...readwriter.DataFrameReader object ...>

Specify the option ‘nullValue’ and ‘header’ with reading a CSV file.

>>> import tempfile
>>> with tempfile.TemporaryDirectory(prefix="options") as d:
...     # Write a DataFrame into a CSV file with a header.
...     df = spark.createDataFrame([{"age": 100, "name": "Hyukjin Kwon"}])
...     df.write.option("header", True).mode("overwrite").format("csv").save(d)
...
...     # Read the CSV file as a DataFrame with 'nullValue' option set to 'Hyukjin Kwon',
...     # and 'header' option set to `True`.
...     spark.read.options(
...         nullValue="Hyukjin Kwon",
...         header=True
...     ).format('csv').load(d).show()
+---+----+
|age|name|
+---+----+
|100|NULL|
+---+----+