Extracting Information from Data

Data, Metadata, and Information

Data are the raw values a computer stores. Information is the meaning we get when we find patterns in data. A list of 10,000 temperatures is data. "July is the hottest month in Phoenix" is information.

When you take a photo with your smartphone, the picture is the data. The phone also saves metadata with it:

Metadata can be useful. You can sort photos by date, search a map for where a picture was taken, or find all songs by one artist. But metadata can also reveal private details. Someone who gets a photo with GPS metadata could learn where you live or hang out. That is why many apps strip location data before you post.

Finding Patterns: Correlation and Causation

Computers are great at spotting patterns in huge datasets. One common pattern is correlation: two variables tend to change together.

• Positive correlation: when one goes up, the other tends to go up.

• Negative correlation: when one goes up, the other tends to go down.

• No correlation: no clear relationship.

Look at Jayden's data: more hours studied, higher quiz score.

The points climb almost along a straight line, so this is a strong positive correlation. A pattern like this lets us make a prediction. The scores go up about 5.75 points per hour, so 6 hours might give about 91 (85 + 5.75 ≈ 90.75). But predictions outside the data you collected (here, beyond 5 hours) are risky. Scores can't go past 100, and extra hours might help less and less.

To say something causes something else, you usually need a carefully designed experiment, not just a pattern.

Filtering and Combining Data

Programs often pull out only the rows they need. Filtering keeps the rows that meet a condition. Grouping puts rows into categories so you can count or average them.

Here is a filtering example like the bus-rider data in the practice exercises, but with a different group:

Data Quality and Bias

Any conclusion is only as good as the data behind it. Two main things can go wrong.

Dirty data

Real datasets are messy. Before a program analyzes data, you clean it. Common problems:

• Inconsistent formats: "TX", "Texas", and "tx" all mean the same state, but a program sees three different values.

• Missing values: blank cells.

• Duplicates: the same person listed twice, which inflates counts.

• Wrong values: an age of 250, or text where a number belongs.

A program that counts rows equal to exactly "TX" would miss "Texas" and "tx" and give a count that is too low. Cleaning means standardizing ("Texas" and "tx" become "TX"), removing duplicates, and deciding how to handle blanks (follow up, or leave them out and say so).

Biased data

Data is biased when it doesn't fairly represent the group you care about. Two common causes:

• Sampling bias: the people you collected from are not typical. A survey sent only to AP students doesn't speak for the whole school.

• Non-response bias: people who reply differ from those who don't. If only 40% reply, the other 60% are missing, and they may feel differently.

A better approach is a random sample: choose students by chance from the full school list, and follow up to raise the response rate.

Using Programs with Large Datasets

Some datasets are far too big to read by hand: millions of rows of trips, posts, or purchases. Programs make it possible to:

• Clean thousands of rows automatically.

• Filter and group to answer a question (for example, trips per hour).

• Visualize the results with charts and graphs so patterns are easier to see.

A program that works on 100 rows should also work on 100 million. We call this scalability: the ability to keep working as the data grows. Large datasets can reveal patterns that small ones can't, but they also raise concerns about privacy (who has your data?) and about whether the data is representative. A giant dataset can still be biased.

Back to the start: the app's claim about walking and sleep is a correlation. To trust it, check how the data were collected, whether it is a fair sample, and whether something else could explain both.

Summary