JSON is now the standard web APIs, configuration files, and data transfer format in almost all programming languages. Python has built-in JSON libraries that help developers read, parse, change, and write JSON data quickly.
This detailed 2600+ word guide will help you understand how to work well with JSON in Python. It includes:
- Statistics on how much JSON is being used and adopted in different industries.
- Comparing tradeoffs – JSON vs XML, CSV, and other formats.
- Understanding and working with complex JSON files – Changing Python objects to JSON format and back – Methods for checking data structure, handling dates, and more
- JSON performance tests for different programming languages – Suggested Python libraries to work with JSON easily
You will find examples of web scraping and automation code showing how to use JSON for everyday scripting tasks.
The Rapid Rise of JSON in Different Industries
JSON began over twenty years ago as part of the JavaScript programming language, but it has grown much larger than that.
Studies on global API traffic indicate that about 70% of all APIs use JSON to send data. Other public web service API surveys found that about 90-95% use JSON instead of different formats.
JSON is popular because it has important benefits compared to older formats like XML and CSV:
- Machine and human readable – JSON is easier to understand than XML because it uses a simpler format but can still represent complex data in a structured way.
- Lightweight—With less syntax overhead, JSON payloads reduce packet sizes, lowering bandwidth demands compared to bloated XML documents.
- Language Agnostic – Because it comes from JavaScript, it works well with JavaScript. However, the text format can easily be used with all modern programming languages.
Let’s compare JSON to other data formats to see some more reasons why it is widely used in areas like web APIs and app configuration files.
JSON vs XML
Ways to organize hierarchical data that can be used in different programming languages.
Benefits of JSON:
- Simplified wording makes it easier to read
- Smaller sizes for quicker data transfer
- Programs can process it quickly and easily
- Works directly with JavaScript
Benefits of XML:
- A wide range of tools developed over many years
- Improved support for namespaces
- Commonly used legacy systems in businesses
YAML is a popular data format similar to JSON but with an even simpler syntax. JSON is still much more popular than YAML because it works well with JavaScript and web applications.
JSON and CSV are two different ways to format data.
JSON (JavaScript Object Notation): This format is often used for web data. It structures data in a way that is easy for computers to read and write. JSON is
Text formats with clear boundaries are often used for tables of data.
Benefits of JSON:
- Can show complex data connections
- It has a flexible structure that doesn’t follow a strict layout
- Works efficiently with programming code
Benefits of CSV:
- A simple format that works with standard tools
- It is more straightforward to view in spreadsheets
- Can read and write data one row at a time
JSON vs Pickle
Lightweight data serialization formats
Benefits of JSON:
- Text that people can easily read vs. binary code
- It can be used with any language
- Much more commonly used
Benefits of Pickles:
- Smaller byte streams.
- Serializes Python-specific constructs
- Quicker performance.
While other options have specific uses, JSON is the best choice overall because it is flexible, simple, compatible, and has good tools. These combined qualities are ideal for the data-focused systems standard in today’s connected programming world.
Next, explain how that flexibility helps use JSON in Python apps.
How to Read JSON in Python
Python’s JSON module is the primary tool for working with JSON. It allows you to encode, decode, read, write, and process JSON data. This section explains how to load JSON data from strings and files to create Python dictionaries, lists, and other structures.
JSON Object Model Mapping
JSON formats easily connect with Python types.
JSON types usually translate easily into standard Python types and vice versa.
Let’s check out some examples of loading that sample JSON document from earlier:
import json
user_json = ‘‘‘
{
"name": "John",
"age": 30,
"addresses":[
{
"street":"123 Main St",
"city":"Anytown",
"state":"CA"
},
{
"street":"456 Broadway",
"city":"New York",
"state":"NY"
}
],
"weekly_schedule":[
{"morning": "Write Code"},
{"morning": "Meetings"}
]
}
‘‘‘
user = json.loads(user_json)This creates a nested dictionary and list structure with the JSON data filled into standard Python objects.
Read JSON from a file.
Loading JSON from a file works similarly to using JSON.load():
with open("user_data.json") as f:
user = json.load(f)For web scraping scripts, you‘ll commonly load JSON results cached from an API request:
import requests
resp = requests.get("https://api.site.com/users")
with open("api_cache.json," "w") as f:
f.write(resp.text)
with open("api_cache.json") as f:
api_data = json.load(f)This saves the API response to a file, so you can rebuild scraped datasets without repeatedly calling the API.
How to Read and Work with JSON in Python
With JSON loaded as Python dict and list structures, all native syntax works for accessing nested elements:
user = json.loads(user_json)
print(user["name"]) # John
print(user["addresses"][0]["city"]) #AnyTown
for address in user["addresses"]:
print(address["street"])Since JSON objects turn into dictionaries, you can use expressions to pull out specific fields and put them into new structures.
name, age = user["name"], user["age"]
addresses = [addr["street"] for addr in user["addresses"]]
Scalar Value Extraction
The jsonpath library makes it easy to find single values quickly.
from jsonpath import jsonpath
street = jsonpath(user, "$..addresses[0].street")
# [‘123 Main St‘]This makes it easier to get specific JSON attributes without having to write complicated code.
How to Convert Python Objects to JSON
The json module converts JSON data into Python objects, including json.dumps() and json.dump() methods turn Python objects back into JSON format.
Convert a Python object to a JSON string.
Convert dictionaries, lists, and other builtin types into a JSON string like: import json
person_dict = {"name": "John", age": 30}
person_json = json.dumps(person_dict)
print(person_json)
‘{"name": "John", "age": 30}‘The serialized JSON string moves data between systems and programming languages, such as web applications, JavaScript, and servers.
Convert a Python object to a JSON file.
To save JSON data to a file, you must provide a file object to the `json.dump()` function.
data = [{"x": 1, "y": 2}, {"x": 3, "y": 4}]
with open("data.json", "w") as f:
json.dump(data, f)Gather data step by step using scripts or web scraping, and regularly save the results as JSON files.
Dealing with Complicated Situations
So far, the JSON shown has been relatively simple. Real projects must often work with tricky data formats, unique objects, special settings, and other complexities.
Let’s go over some everyday unusual situations you might face.
Check the Schemas.
APIs and scraped data can create incorrect JSON, which must be checked for errors before use.
Incorrect:
{
"name": "John"
"age" 30
}
Valid:
{
"name": "John",
"age": 30
}Use a schema-validator like jsonschema to catch format issues:
from jsonschema import validate
user_schema = {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"}
}
}# Raises jsonschema.exceptions.ValidationError if invalid validate(user_json, user_schema)
This will work, but it will still not succeed if the standards are not available. Defensive code should wrap json.loads() in a try/catch as well.
Change to Custom Objects.
Instead of using a regular dictionary, you might want to convert JSON directly into your class objects.
Class User:
def __init__(self, name, age):
self.name = name
self.age = age
# Supply custom object_hook
user = json.loads(user_json, object_hook=lambda d: User(d["name"], d["age"]))This function tells the json module how to deserialize using your custom class.
Work with Dates and Times.
Standard JSON can’t handle dates, so they lose timezone information when they are converted to strings.
{
"now": "2023-02-21T12:34:28.224Z"
}Implement custom decoders to convert timestamps back to datetime on parse:
from datetime import datetime
import dateutil.parser
def json_datetime_hook(json_dict):
for key, value in json_dict.items():
try:
json_dict[key] = dateutil.parser.parse(value)
except:
pass
return json_dict
data = json.loads(data_json, object_hook=json_datetime_hook)
print(data["now"]) # 2023-02-21 12:34:28.224000+00:00JSON formats are designed to be general and work for many purposes rather than focusing on special types like dates or custom objects. Python makes it easy to add extra features that suit your specific needs.
JSON Performance in Different Languages
Python usually values clear and easy-to-read code over speed. However, when dealing with huge JSON files, you need faster processing times for encoding and decoding.
Here‘s how popular languages stack up parsing a 3.2 MB JSON file:
C compilers like GCC and languages like Rust or Go, more orientated for speed, demonstrate 3-4x faster parse times.
Python is still a strong choice because it makes it easy to work on exploratory data tasks. For pure performance, Python alternatives like ujson also accelerate JSON handling further.
Python is great for working with JSON in most situations because it is easy to use, works well with other tools, and is fast.
Conclusion
JSON will not replace formats designed for specific purposes, like Protobuf for space efficiency, CSV for tabular data, or Avro for complex data types. Its flexibility and popularity make it a useful and lightweight common language that can be used in almost any project involving data.
