A paradox of big data
Kaiser tries the new subway card.
One of the challenges in the age of big data is having too much information. Information paralysis. Not enough time to look at all the data. Too many contradictions in the data. And so on.
***
But there is something else that's been bothering me. The more information is collected, the less data we have at our disposal. This is a paradox of the big data age. What many of us face is not actually too much information but not enough data, and not the right data.
I have many examples of this paradox. Let me focus on one recent experience.
Recently, I tried using the OMNY card to ride the New York subway. This is a new card technology that will eventually replace the iconic swipe cards. It's a chip card that you place on a screen and if ithe transaction succeeds, the screen's border lights up green and you walk through the turnstile. The first time I used it, the green light didn't come on so I attempted again, and then the green light appeared.
At that moment, I realized that I wasn't sure if I was charged once or twice. Do I know fore sure that the first transaction didn't succeed? Other than the green signal, the new technology gave me zero information.
Contrast this with the existing swipe cards: for every successful swipe, the display indicates how much was charged, and how much stored value still remains in the card. The former confirms the transaction while the latter helps me plan when to refill the card's value store.
With the OMNY tech, I just have to "trust" the system.
The next day, I transferred from the train to the bus. According to the rules, the second leg should be free, since the transfer happened within a few minutes. With existing swipe cards, the display confirms immediately that I have used a free transfer.
With OMNY, I had no idea: the screen sent the same signal – green light – as for a paid trip.
I later checked the stored value on the OMNY card. I was indeed charged twice on that first trip while the transfer was not charged as expected.
***
What's the paradox? A lot more data are being collected by new tech, and yet users are provided less information than before.
This isn't hard to explain though. Businesses are aware that information has competitive value, and they are hoarding the data, even hiding them from the data's owners (you and I).
