Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

edsu/twarc

a command line tool for archiving JSON twitter search results before they disappear

People

Repo info

Activity

19:22

edsu commented #498
19:18

edsu on main

added --text and --csv output t… (compare)
19:06

igorbrigadir on counts

(compare)
19:05

igorbrigadir on main

add counts unit test Merge pull request #498 from Do… (compare)
19:05

igorbrigadir closed #498
19:05

igorbrigadir opened #498
19:03

igorbrigadir on counts

add counts unit test (compare)
18:54

edsu on main

add suport for new counts endpo… remove max_results from counts … format & remove expansions para… and 6 more (compare)
18:54

edsu closed #497
18:52

edsu commented #497
18:51

igorbrigadir opened #497
18:51

igorbrigadir on counts

remove uneused max_results para… (compare)
18:40

igorbrigadir on counts

fix up call and help messages (compare)
18:28

igorbrigadir on counts

refactor expansions fix broken `twarc2 following` c… refactor ensure flattened and 23 more (compare)
18:27

igorbrigadir on counts

add suport for new counts endpo… remove max_results from counts … format & remove expansions para… and 2 more (compare)
16:02

AbirRes opened #496
12:26

edsu on v2.2.1

(compare)
12:25

edsu on main

new version (compare)
11:00

edsu commented #495
10:56

edsu on main

Handle quoted user ids in twarc… Don't just pass through arbitra… added some logging and 1 more (compare)

Ed Summers

@edsu

Renato Gabriele

@remagio

so using pipe it can be checked

Ed Summers

@edsu

zcat is nice on linux

zcat file.gz | grep foo

Renato Gabriele

@remagio

not yes used in bash scripts

Ed Summers

@edsu

but i'll leave it uncompressed for now

Renato Gabriele

@remagio

not yet

Ed Summers

@edsu

Renato Gabriele

@remagio

but it could be when you think it's fine

i'm discussing with founders of http://www.journalismfestival.com how to approach some sessions about socials&politics

Ed Summers

@edsu

very nice

Renato Gabriele

@remagio

My last speech at HackMeeting14 was about how happened some dadaist performances on socials during last national and european election. Showing how some kinda "blacktop" works on socials.

Ed Summers

@edsu

dada++

Renato Gabriele

@remagio

"blackop"

Ed Summers

@edsu

what is "blacktop"?

Renato Gabriele

@remagio

blackop

Raised a fact, during previous election huge CnC where manipulation discussions

Specially during live TV events with candidates.

The dadaist performances planned to contrast and win.

it raised and showed up during live TV shows.

sorry "were" not "where" f**k automatic correction

Renato Gabriele

@remagio

(political smm+devs committees working like usually black-op does)

Bye @edsu, ttys

Ed Summers

@edsu

@remagio ok, give it a try utils/archive.py

faustusdotbe

@faustusdotbe

Hello

I'm new here (first time ever I'm using gitter.im)

I have a small problem with twarc, not sure if I'm doing something wrong or if it's not finished yet: every I try to used twarc.py (or as a library), I get this error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "twarc.py", line 114, in init
raise argparse.ArgumentTypeError("Please make sure CONSUMER_KEY, CONSUMER_SECRET ACCESS_TOKEN and ACCESS_TOKEN_SECRET environment variables are set.")
argparse.ArgumentTypeError: Please make sure CONSUMER_KEY, CONSUMER_SECRET ACCESS_TOKEN and ACCESS_TOKEN_SECRET environment variables are set.

is it normal?

(I have edited twarc.py with correct keys from an app I created at dev.twitter.com

)

faustusdotbe

@faustusdotbe

Okay I forgot to set the environment variables, my bad...

Ed Summers

@edsu

I'm glad you figured it out ; since it took so long to respond. You can use command line parameters now too if you want.

edsu @edsu waves

Ed Summers

@edsu

v0.3.4 is out now ; only change was moving utils/archive.py to utils/twarc-archive.py and now installing it as part of pip install.

nbaur

@nbaur

Hi Ed finally got my twarc-archive.py script running with the following command line: nataliebaur$ twarc-archive.py semanaNT ~/semanaNT. apparently needed the little squiggly thing in front of the directory structure.

Ed Summers

@edsu

@nbaur The tilde is just shorthand for your home directory :-) So I guess you have a folder semanaNT in your home directory?

nbaur

@nbaur

@edsu yes it's in my home directory. i'm still learning the basics of command line :)

Ed Summers

@edsu

ok, definitely let me know of questions as they come up ; i expect you will want to schedule it to run periodically, or are you just going to run it manually?

nbaur

@nbaur

another question. i want to run a search for the string of words "congreso patrimonio nuevas tecnologias" but when I run the twarc-archive.py script for that I get the error: archive: error: unrecognized arguments: nuevas tecnologias /Users/nataliebaur/mexicoconfs_twitter/pcnt

nbaur

@nbaur

Hi @edsu Natalie here again. Question about wordcloud util. I ran command: ~/twarc-master/utils/wordcloud.py ~/mexicoconfs_twitter/ciads_unam/tweets-0001.json > ~/mexicoconfs_twitter/ciads_unam/wordcloud.html
but when i open wordcloud.html it is empty.

Ed Summers

@edsu

@nbaur i just saw your question! (oops)

Screen Shot 2015-12-08 at 3.55.24 PM.png

@nbaur: i just tried it, and it worked for me

feel free to send me the .json file via email or something if you want me to try

Steven Brien

@spbrien

Hello all. Is anyone having issues with 400 Bad request errors from the twitter API? All of a sudden I can't seem to get a successful response and was wondering if I am alone on this.

Steven Brien

@spbrien

Nevermind it's an issue with my twitter account. BTW @edsu this is an awesome tool. I've been using it with mongodb and implementing further filtering/analysis of saved tweets through the database. Working on a JSON API/web front end.

Renato Gabriele

@remagio

Not sure if I'm right, but I still remember that sometimes ago while I was looking with stream&archive for "#some" it was able to get also results with "#something" "#someothers" and so on... am I wrong?

Renato Gabriele

@remagio

hi there, still thinking about my previous question but I'm now on a new feature of twarc since availability of -w / --warnings option

since it place warnings in the output json it brake utils or other python scripts I wrote, anyone tried out to address it? Not sure if better cleanup warning at every output of json stream or alter twarc src to keep saving warnings outside the default json stream

just to say, since the beginning of -w / --warnings I got only with ONE hashtag analyzed, so I have no idea about the real convenience in managing it