Support NeoGAF

[QuantumBro]

Regex

Scrape the HTML off of every page and then search for "vote"

That was what I was thinking, probably start out just by grabbing the vote count and then when I have more time, be able to post the names of who voted.

You would program a page scraper using Python or something, give it the pages you want to look at, and then use (as Makai said) regex to count occurrences of votes and unvotes and which posters are associated with them by looking through each page's source code.

Don't ask me how to do this, though. I know the theory, not the implementation.

Yeah I'm going with python, just took three lines of code to scrap a page and output it.

Actually, you could search for the highlight tags if you can get access to the markup.

/([A-Z])\w+/g

That's what I'm thinking since I'll also have to find the unvotes.
Highlight tags are <span class="highlight"></span>

 

[Odrion]

Mafia Werewolves?! Fucking aw-*reads the thread*-oooooh. ._.

 

[Haly]

You're looking for a pattern like:

<!-- post #166373097 -->
<a href="member.php?u=93188">Makai</a>
<span class="highlight">VOTE: Crab</span>
<!-- / post #166373097 -->

 

[johnnyquicknives]

I was thinking about trying to put together an app yesterday that would keep the current vote total at the top of the page. Turned out I was too lazy to actually do it though.

Remember to strip the blockquotes from posts before grabbing the votes.

 

[QuantumBro]

I should have some free time this weekend, so I'll write up a prototype and put it up on github.

You're looking for a pattern like:

<!-- post #166373097 -->
<a href="member.php?u=93188">Makai</a>
<span class="highlight">VOTE: Crab</span>
<!-- / post #166373097 -->

Also have to look at the post number to make sure it doesn't count posts from previous days and knows what pages to look through.

I was thinking about trying to put together an app yesterday that would keep the current vote total at the top of the page. Turned out I was too lazy to actually do it though.

Remember to strip the blockquotes from posts before grabbing the votes.

Totally forgot about quotes.

 

[johnnyquicknives]

Totally forgot about quotes.

yeah it was one of the things I thought of when brainstorming it yesterday. I don't really use python much but if its on git and I can help out a bit I will.

When I've used python for scraping before I've used pyquery. It will be more verbose but you can use jQuery style selectors which I prefer to regex.

 

[johnnyquicknives]

Oh and one of the other gotchas I noticed is that gaf will let you continually increment page number but always show the last page:
http://www.neogaf.com/forum/showthread.php?t=953137&page=47
http://www.neogaf.com/forum/showthread.php?t=953137&page=10

so you'll probably need to track post numbers if you're scanning the whole thread. I'd imagine for days you can look at the user id of the OP and then check any later posts by them for the start of day blurb.

 

[QuantumBro]

Oh and one of the other gotchas I noticed is that gaf will let you continually increment page number but always show the last page:
http://www.neogaf.com/forum/showthread.php?t=953137&page=47
http://www.neogaf.com/forum/showthread.php?t=953137&page=10

so you'll probably need to track post numbers if you're scanning the whole thread. I'd imagine for days you can look at the user id of the OP and then check any later posts by them for the start of day blurb.

Yeah I was thinking of having two variable, one being the number of the first post of the day, and one being the number of the last post made in the thread, and then dividing those numbers by 50 and rounding the resulting numbers up to figure out the range of pages to go through.

 

[johnnyquicknives]

Yeah I was thinking of having two variable, one being the number of the first post of the day, and one being the number of the last post made in the thread, and then dividing those numbers by 50 and rounding the resulting numbers up to figure out the range of pages to go through.

Should work, post numbers are a bit of a pain to grab.
I'm sure this is probably horrible to someone more familiar with python but this is a snippet of code I had lying around from when I was scraping allrecipes. It would give an idea of the pyquery syntax which I found pretty good for scraping.

import requests
import json
import urllib
from time import sleep
from pyquery import PyQuery as pq

BASE_URL = "http://www.allrecipes.com"

def get_recipes(ingredient, quantity):
    results = []
    total_results = False
    page_number = 1
    
    while len(results) < quantity:
        print "searching for %s. Page %i. %r Results found." % (ingredient, page_number, total_results)
        # grab the page using requests
        response = requests.get(BASE_URL + "/search/default.aspx?w0=" + urllib.quote_plus(ingredient) + "&Page=" + str(page_number))
        # encode page for pyquery
        d = pq(response.content)
        
        if not total_results:
            total_results = int(d(".searchResultsCount .emphasized").text().split(" ")[0].replace(",", ""))
            if total_results < quantity:
                quantity = total_results
        
        for recipe in d(".grid-result-cntnr").items():
            if len(results) < quantity:
                results.append(BASE_URL + recipe(".recipe-info .title").attr("href"))
                
        page_number += 1
        sleep(2)
    
    return {"ingredient": ingredient,
            "recipes": results}

 

[Karkador]

Wow this is awesome

I've been making the vote tallies show who votes for someone, and then pulled out of the vote. That was a feature that felt needed in the first game, especially after many weeks and thousands of posts to comb through.

 

[EzekelRAGE]

http://wiki.mafiascum.net/index.php?title=Anarchy
These roles look fun as hell:
-The Anarchist visits two people per night and gives the visited players a bomb. The bomb explodes after 2 days and the player holding it dies.
-Each night the Arsonist visits one player, that player is then fused. Once per game instead of fusing somebody he can blow up all fuse players. After that he looses his ability.
-The Day Suicide Bomber can PM the mod to kill himself and one other player during the day.

Poisoners sound like fun as well. Hope we can get a variation or two of these next games.

 

[Ourobolus]

Wow this is awesome

I've been making the vote tallies show who votes for someone, and then pulled out of the vote. That was a feature that felt needed in the first game, especially after many weeks and thousands of posts to comb through.

Are you using some sort of program to calculate votes, or is it manual?

 

[Karkador]

Are you using some sort of program to calculate votes, or is it manual?

Manual :/

 

[Haly]

Aren't you glad we're not as active as the Star Wars thread?

MattyG must be going crazy from tallying up votes.

 

[Karkador]

Haha, yes!

 

[raindoc]

Seriously, can someone check on Raindoc?

 

[Zatoth]

 

[tomakasatnav]

Poor Barry, guy can't catch a break.

 

[Deleted member 231381]

LunaticLMD, followed by The Wizard, followed by squidyj, followed by Hagi, are the next three people on the waiting list, MattyG. Contact them in that order. If you have to, extend this night until you get a positive response. Annoying for the players, but it's more disruptive to have a replacement in halfway through a day.

 

[tomakasatnav]

Lunatic is already in the Animal Crossing game (and not active there either!)

 

[Deleted member 231381]

Lunatic is already in the Animal Crossing game (and not active there either!)

Well-spotted, I didn't realize Lunatic had been replaced in. I'll find the new third in case MattyG needs it.

 

[YesNOnoNOYes]

Pick me, pick meeeeee

I'm super active~ and very talky &#128515;

 

[Zatoth]

Pick me, pick meeeeee

I'm super active~ and very talky &#128515;

You'd probably be just another Blarg. Picking up your chrome spray and go straight to Valhalla, no matter your role.

 

[YesNOnoNOYes]

You'd probably be just another Blarg. Picking up your chrome spray and go straight to Valhalla, no matter your role.

XD

He did die historic

But nu! I will try to be smart and #win and stuff!!! I'll try to sniff all the hutts with my

Spoiler

nonexistant

wiles and charms!!! I'll be like super useful and stuff~

Spoiler

at confusing others and myself

o7

 

[raindoc]

XD

He did die historic

But nu! I will try to be smart and #win and stuff!!! I'll try to sniff all the hutts with my

Spoiler

nonexistant

wiles and charms!!! I'll be like super useful and stuff~

Spoiler

at confusing others and myself

o7

 

[YesNOnoNOYes]

>

I'm working my magic already!!!

Even before I'm in the game!!!

 

[Blargonaut]

You'd probably be just another Blarg. Picking up your chrome spray and go straight to Valhalla, no matter your role.

I WAS THE SCALES OF JUSTICE

DETAIN, PLAYERS, DETAIN

 

[Ourobolus]

I'm still going over the last few pages of the Star Wars thread and just laughing my ass off. It's just...madness.

 

[Zubz]

LunaticLMD, followed by The Wizard, followed by squidyj, followed by Hagi, are the next three people on the waiting list, MattyG. Contact them in that order. If you have to, extend this night until you get a positive response. Annoying for the players, but it's more disruptive to have a replacement in halfway through a day.

We're getting a Wizard? Or better yet, THE Wizard? Those Wolves are so screwed!

If you really can't wait for a new Blarg that badly, I could take over the role for you!

 

[Zatoth]

The Animal Crossing thread needs way more crazy. They play it like decent human beings. Those monsters.

 

[Palmer_v1]

The Animal Crossing thread needs way more crazy. They play it like decent human beings. Those monsters.

I assume they're our descendants, who have slowly become more civilized. When they dig up fossils in their free time, they should check for blaster marks.

 

[Haly]

Animal Crossing thread externally:

Animal Crossing thread internally:

 

[Kevyt]

Wow... what's up with all that code? I thought I clicked on the wrong thread for a minute, lol.

That animal crossing mafia thread sure is entertaining!

 

[Zatoth]

Animal Crossing thread externally:

Animal Crossing thread internally:

Every moment in the Star Wars game (internally and externally):
https://www.youtube.com/watch?v=NZaG_13ZIYY

 

[El Topo]

The Animal Crossing thread needs way more crazy. They play it like decent human beings. Those monsters.

It reminds me of some X-Files or Outer Limits episode. A nice suburban neighborhood, with unspeakable terror and dark secrets hidden. Maybe we learn in the end that they're all cannibals. Those that left didn't get "evicted".

 

[Palmer_v1]

I will laugh heartily if they ended with a tie, i.e. No Eviction.

We get shit done in the Star Wars thread, at least.

 

[RobotNinjaHornets]

It reminds me of some X-Files or Outer Limits episode. A nice suburban neighborhood, with unspeakable terror and dark secrets hidden. Maybe we learn in the end that they're all cannibals. Those that left didn't get "evicted".

Holy shit, we are actually doing Twin Peaks mafia secretly

 

[Haly]

I will laugh heartily if they ended with a tie, i.e. No Eviction.

We get shit done in the Star Wars thread, at least.

Yes, so much shit done that sometimes I wonder if most of you are Hutt rather than Rebel.

 

[Palmer_v1]

Holy shit, we are actually doing Twin Peaks mafia secretly

lol, I preferred The Burbs references someone made before. You need a little Tom Hanks silliness to lighten the mood.

 

[Palmer_v1]

Yes, so much shit down that sometimes I wonder if most of you are Hutt rather than Rebel.

Maybe we're all that role that always has to vote yes on lynches.

 

[LaunchpadMcQ]

I would be interested in running a Danganronpa themed game, if that's a theme people would enjoy.

 

[Deleted member 231381]

I will laugh heartily if they ended with a tie, i.e. No Eviction.

We get shit done in the Star Wars thread, at least.

I mean, if "getting shit done" means lynching all your own power roles, yes, you get shit done.

 

[Razmos]

I would be interested in running a Danganronpa themed game, if that's a theme people would enjoy.

Oh my god yes.
Hope vs Despair, so cool.

 

[squidyj]

I would be interested in running a Danganronpa themed game, if that's a theme people would enjoy.

I don't know about Danganronpa or Animal Crossing, like, at all. I don't mind themes I don't get but it's more fun the more people who are going to be 'in on it' yknow?

 

[LaunchpadMcQ]

Oh my god yes.
Hope vs Despair, so cool.

I've got what I think are some funny Ultimates in mind

I don't know about Danganronpa or Animal Crossing, like, at all. I don't mind themes I don't get but it's more fun the more people who are going to be 'in on it' yknow?

That's why I asked, to gauge interest

 

[Haly]

I toyed with the idea of a cyberpunk themed Mafia game where the major mechanic is an anonymous, public board all Town-aligned can post in once per day.

 

[AbsolutBro]

Yes, so much shit down that sometimes I wonder if most of you are Hutt rather than Rebel.

22 rebels, 1 serial killer.

 

[Palmer_v1]

I mean, if "getting shit done" means lynching all your own power roles, yes, you get shit done.

Admit it. You're just frightened that you might get teamed up with some of our brilliant tacticians next game!

I demand t-shits be prepared for the eventual winners that simply say, "I survived the Star Wars"

Edit:

I don't know about Danganronpa or Animal Crossing, like, at all. I don't mind themes I don't get but it's more fun the more people who are going to be 'in on it' yknow?

I generally prefer the standard mafia or Werewolf themes because they're pretty universal. It's part of why I still break things down to Town vs Mafia rather than Rebels vs Hutts in our game.

 

[Deleted member 231381]

I demand t-shits be prepared for the eventual winners that simply say, "I survived the Star Wars"

"t-shits" would be a fitting tribute.

Spoiler

<3 u

 

[Darryl]

I toyed with the idea of a cyberpunk themed Mafia game where the major mechanic is an anonymous, public board all Town-aligned can post in once per day.

I would have a field day

Animal Crossing thread externally:

Animal Crossing thread internally:

We are just a bunch of gossips