Using twarc to pull in the dataset

[igorbrigadir]

WIP

[igorbrigadir]

Created rt_queries.txt from

csvcut -c screen_name congressmember_data.csv > screen_names.txt

First, getting counts of all the data with:

twarc2 searches --archive --start-time "2021-01-01" --end-time "2021-10-23" --counts-only --combine-queries --granularity "day" rt_queries.txt rt_queries_counts.csv

Then checked the total tweets with:

import pandas as pd
df =pd.read_csv("rt_queries_counts.csv")
df["day_count"].sum()

And it was 41563787 which would take too long to download all of them with the Monthly tweet cap restrictions. Unfortunate.

However, i'd like to use this as a motivating example for DocNow/twarc#566 later, so i'll update the PR here with the new results when i get them!

[igorbrigadir]

rt_queries_counts.csv are counts of combined search queries but maybe it will be more useful to see RT counts of each individual, so i'll add individual_day_counts.csv shortly