How to Scrape 1 Million Tweets — Without the Twitter API
We scraped 1 million tweets that mentioned the hashtag ‘Islam’ in about 12 hours.
Why Twitter Matters
Twitter is one of the most important social media platforms out there used by individuals, companies, activists, media, and governments. In fact, some would argue it’s the world’s defacto town square (Wall Street Journal 2022). Twitter played a pivotal role in the unfolding of world-changing events such as the Arab Spring, Black Lives Matter, #MeToo, controversial elections, and the COVID-19 Pandemic.
The micro-blogging approach allows people to send short messages forcing them to focus on concision and quality, allowing for ideas to spread quickly through highly dense networks. It’s a great way to connect with like-minded individuals and form communities in ways that other platforms like Facebook, Youtube, and perhaps even Reddit lack.
Every second, on average, around 6,000 tweets are tweeted on Twitter […], which corresponds to over 350,000 tweets sent per minute, 500 million tweets per day and around 200 billion tweets per year. (Internet Live Stats)
With so much information being generated and its ability to provide important insights on humanity, analyzing (and therefore…
