Mestiv's signature

[the winter system]

26 minutes ago, Mestiv said:

Like someone has to click on it to let it know it's OK to track this user? That's possible I think, but I don't know when I''d have the time to implement it. 

Yeah, thanks. When you have the time, preferably soon. Like when I saw it, I thought everyone was seeing my information blasted all out of the internet. Which was weird. 

[TwiLyghtSansSparkles]

15 minutes ago, winter devotion said:

Yeah, thanks. When you have the time, preferably soon. Like when I saw it, I thought everyone was seeing my information blasted all out of the internet. Which was weird. 

It's totally secure, though. A little disconcerting to look at, but it doesn't store the information at all. 

[the winter system]

1 minute ago, TwiLyghtSansSparkles said:

It's totally secure, though. A little disconcerting to look at, but it doesn't store the information at all. 

Okay, good. Mestiv can't even look if he wants to?

[Mestiv]

1 minute ago, TwiLyghtSansSparkles said:

It's totally secure, though. A little disconcerting to look at, but it doesn't store the information at all. 

Yeah, to make everyone feel safer  I can release the code behind it, so other programmers can confirm how it works. 

[Mestiv]

2 minutes ago, winter devotion said:

Okay, good. Mestiv can't even look if he wants to?

To be honest I would be able to take a peek at location guesses my program does if I wanted to, but I'm not storing this info anywhere and I have no interest  in it. Also as I have mentioned before in this topic and many users can confirm, the location guessing is very inaccurate. 

 

I will publish the code behind my signature tomorrow, so it will be 100% clear what it can do. If you have any questions about it I'll be happy to answer and I'll make it voluntary soon. 

[Oversleep]

The question is: if the information we're all sending everywhere can be easily looked up, is it wrong to do so?

[Darkness Ascendant]

I'm from somewhere near Australia. XD

Anyway, what do u guys mean "it learned your name". I only use my laptop for the Shard (mostly). And I've seen Mestiv's sig loads of times. But it still calls me "fellow sharder"

[Mestiv]

4 minutes ago, Darkness Ascendant said:

I'm from somewhere near Australia. XD

Anyway, what do u guys mean "it learned your name". I only use my laptop for the Shard (mostly). And I've seen Mestiv's sig loads of times. But it still calls me "fellow sharder"

That's strange it didn't ever display your name. It should have already recognized you... Try refreshing a page with my signature a couple more times. It's possible that your browser is configured in such way, that my signature forgets you every time you shut it down.

---

Ok, I promised to publish the code responsible for my signature, so here it is. At least, the important part of the code. I'm not going to post server configuration files or anything like that. So, here are the important functions that take part in creation of my signature. get_svg_signature() is the entry point (at the bottom). Lines starting with # are comments I added to make it easier to understand.

 

Spoiler

def get_ip_location(ip):
    """
    Return location guess based on the IP address.
    Using GeoIP2 database.
    """
    global geo_reader
    t = time.time()
    location = geo_reader.city(ip)
    country = location.country.name
    country_iso = location.country.iso_code
    subdiv = location.subdivisions.most_specific.name
    city = location.city.name
    if city is None:
        return country
    if country_iso == 'US':
        return "{} ({}, {})".format(city, subdiv, country)
    return "{} ({})".format(city, country)


def get_title_page(url):
    """
    Given URL address of a webpage (assuming it's a thread from 17th Shard),
    discover the title and page number of a thread.
    """
    t = time.time()
    cached_data = cache.get(url)
    if cached_data is not None:
        # If I have this URL in my local memory, I don't need to check it.
        return cached_data
    try:
        # Call another process to download the page.
        html_ = subprocess.check_output(['wget', '-q', '-O', '-', url]).decode()
    except subprocess.CalledProcessError:
        # Something went wrong with downloading the page.
        if url.startswith('http://www.17thshard.com/forum/') and 'do=showConversation' in url:
            # 17th Shard didn't allow to download the page, because it's a private conversation.
            return 'a private conversation', False
        # Something else went wrong. No idea what's behind this URL.
        return 'a page unavailable to anonymous users', False
        
    title_re = '<h1.*?>(.*?)</h1>'
    title_re = '>([^>]+)</h1>'
    page_re = '<a.*?>Page (\d+) of (\d+).*?</a>'
    
    # Using regular expressions to find page title ...
    title = re.search(title_re, html_, re.M | re.DOTALL)
    if title is not None:
        title = title.group(1)
        title = html.unescape(title).strip()
        
    # ... and page number
    page = re.search(page_re, html_, re.M | re.DOTALL)
    if page is not None:
        page = page.group(1).strip()
    else:
        page = 1
    # Keep the title and page number in local memory for future use.
    cache.set(url, (title, page), 600)
    return html.escape(title), page


def gather_id_data(url, session_id):
    """
    Read the online users page and see who was the last person to visit given url.
    
    The user on the 17th Shard user list that last visited given thread is
    given a 'point' associated with given session_id.
    """
    url2 = 'http://www.17thshard.com/forum/online/?filter=filter_loggedin&sortby=running_time'

    url_re = '/forum/topic/(\d+?)-'
    s = re.search(url_re, url)

    if s is None:
        # This URL doesn't point to a forum thread.
        return

    topic_id = s.group(1)

    try:
        # Give the Shard 2 seconds to update the online users list.
        time.sleep(2)
        # Then, download the list.
        html_ = subprocess.check_output(['wget', '-q', '-O', '-', url2]).decode()
    except subprocess.CalledProcessError:
        # If the list is not reachable, abort procedure.
        return

    # Perform some juggling on the HTML code of the online users list to find what I need.
    row_search = '/forum/topic/{}-'.format(topic_id)
    right = html_.find(row_search)
    if right == -1:
        return
    right = html_[:right].rfind('</h3>')
    left = html_[:right].rfind('<h3')
    row = html_[left:right]
    name_re = '<a.*?>(?P<name>.+?)</a>'

    # Use regular expression to find a username.
    name = re.search(name_re, row)
    if name is None:
        # No username found. I guess I was too late to spot who visited given thread.
        return
    name = name.group('name')
    # I have a name of the last person visiting given thread.
    # I save it in my database. SessionData is a class I use to store this information.
    sd = SessionData.objects.get_or_create(session_id=session_id, username_guessed=name)[0]
    sd.number_of_guessess += 1
    sd.save()
    # Database is updated. Someone has been given a point associating him/her with a session_id.
    return
    
def guess_name_for_session(session_id):
    """
    Check if we have enough data about given session in the database to guess a name.
    """
    if session_id is None:
        # None is not a session number.
        return False

    # Get all users associated with this session_id. Ordered by number of points.
    sds = SessionData.objects.filter(session_id=session_id).order_by('-number_of_guessess')
    # Get the sum of all the points related to this session_id.
    total_count = SessionData.objects.filter(session_id=session_id).aggregate(Sum('number_of_guessess'))['number_of_guessess__sum']
    # Get the data entry with highest score.
    most_points = sds.first()
    max_count = most_points.number_of_guessess if most_points is not None else 0

    if max_count > 10 and max_count / total_count > 0.5:
        # If the entry with highest number of points has over 10 points and
        # it has over the half points associated with this session_id
        # I assume that it's correct guess.
        return most_points.username_guessed
    # I don't have enough data to guess what user to associate with this session.
    return False


def get_svg_signature(request):
    try:
        # Check in which thread the signature is going to be displayed.
        # HTTP_REFERED contains URL of the thread.
        url = request.META['HTTP_REFERER']
        title, page = get_title_page(url)  # My function defined above.
    except KeyError:
        # There was no HTTP_REFERED - I cannot know where the signature will be displayed.
        title = None
        page = None
        url = None

    if 'REMOTE_ADDR' in request.META and request.META['REMOTE_ADDR'] == '50.116.10.77':
        # Make sure that if it's the 17th Shard server asking for my signature,
        # send it something that will be accepted :P 
        # (10x10 white square in PNG format).
        img = Image.new('1', (10, 10), 255)
        resp = HttpResponse(content_type='image/png')
        img.save(resp, 'PNG')
        return resp

    # Get session_id associated with the browser.
    session_id = request.session.session_key
    # Guess the username using function defined above.
    name = guess_name_for_session(session_id)

    if session_id and url:
        # If I have session_id and URL address, I can update my database and 
        # give someone a point. I do it in another thread.
        t = threading.Thread(target=gather_id_data, args=(url, session_id))
        t.start()

    # My signature is in SVG format. SVG pictures are stored as text in files.
    # I read a template for my signature from a file.
    svg_text = open('sig_template.svg', mode='rb').read().decode()

    if not name:
        # I couldn't guess who's displaying my signature, so I call them 'fellow Sharder'
       name = 'fellow Sharder'

    # I process the page title and page number I received earlier.
    if title:
        if page:
            if len(title) > 35:
                title = "{}...".format(title[:32])
            topic = "I see you're browsing {} on page {}.".format(title, page)
        else:
            if len(title) > 47:
                title = "{}...".format(title[:44])
            topic = "I see you're browsing {}.".format(title)
    else:
        topic = "I have no idea where are you now :P"

    if 'REMOTE_ADDR' in request.META:
        # If I have the IP address in the request, I try to guess it's location.
        ip = request.META['REMOTE_ADDR']
        loc = get_ip_location(ip) # Defined above
        place = "Ohh, you're from somewhere near {}, right?".format(loc)
    else:
        place = ''

    # Filling my signature template with processed data (topic title, page number and guessed location).
    svg_text = svg_text.format(place=place, name=name, topic=topic)

    # Packing the signature inside HTTP response.
    resp = HttpResponse(content_type='image/svg+xml')
    # I need to set some data associated with given session to make it work.
    request.session['time'] = time.time()
    resp.content = svg_text.encode()
    return resp

 

 

I will obviously have to modify this code to allow users to turn my signature on and off, but the core functionality will stay the same.

[Magestar]

My computer is pretty well protected.  I'm fairly certain that's why, even though I once reloaded the page your Sig directs to 50 times in a row (Just to check, you see.) it still does not have my name, or most other things.  It does get the page, which is somewhat freaky.  And recently, it's gotten close enough to where I live to make me jump when I first saw it.  But, I'm pretty sure that most websites are capable of doing that anyway, such as Facebook, and most online shopping sites, so I don't really see anything wrong with this.  It's just creepy.    Also, as a person who likes to dabble in CS and programming, it's kind of awesome to look at.  Especially now that I can see the source code.

Edited October 3, 2016 by Magestar

[Darkness Ascendant]

Ok, It recognises me now  Took more than a few refreshes however

[Elenion]

Refreshed about 5 times and then it got me. That's creepy and awesome at the same time.

On 10/2/2016 at 3:39 PM, Oversleep said:

The question is: if the information we're all sending everywhere can be easily looked up, is it wrong to do so?

As long as the data isn't of confidential nature or stored without the user's knowledge and permission, I don't see why it would be wrong.

[Ironeyes]

I am completely willing to admit that your sig scared the living daylights out of me when I first saw it. Then I followed the link and read your explanation, and now it seems really cool. But still kind of creepy. 

[Hemalurgic Headshot]

The accomplishments of coding.

[Queen Elsa Steelheart]

It even got my suburb...that's awesomely creepy 

[Steeldancer]

WHAT THE LIVING HECK IS GOING ON???? I JUST REQUESTED THE DESKTOP SITE AND WHATTTTTTT? And besides knowing where I live, what are the songs on it for? I don't know three of them, but I've listened to "wait for it" from Hamilton fairly recently. Which is weird. 

Oh ok it's music YOU'VE listened to recently. Wow that's pretty awesome. Also I glad you like Hamilton 

Edited June 6, 2017 by The Flash

[Mestiv]

18 minutes ago, The Flash said:

WHAT THE LIVING HECK IS GOING ON???? I JUST REQUESTED THE DESKTOP SITE AND WHATTTTTTT? And besides knowing where I live, what are the songs on it for? I don't know three of them, but I've listened to "wait for it" from Hamilton fairly recently. Which is weird. 

Oh ok it's music YOU'VE listened to recently. Wow that's pretty awesome. Also I glad you like Hamilton 

Yeah, don't worry, I don't know what you're listening to

[Steeldancer]

18 hours ago, Mestiv said:

Yeah, don't worry, I don't know what you're listening to

Yeah good thing. Because I listen to my music on a different device 

[Dragon314]

Calls me fellow harder and is off by 1,628 miles

[flying_shadow]

Goddamit, I thought I was losing my mind when it got my location right.

[SilverTiger]

Your signature nearly gave me a heart attack when I first saw it. Which is kind of a compliment, if you wanted to murder startle people.

Edited October 13, 2017 by SilverTiger

[Mestiv]

2 hours ago, SilverTiger said:

Your signature nearly gave me a heart attack when I first saw it. Which is kind of a compliment, if you wanted to murder startle people.

Heart attacks weren't my goal. I just want to surprise people

[Oversleep]

5 hours ago, Mestiv said:

Heart attacks weren't my goal.

Yeah it's much more convenient to do with Death Note. Although usage of signature is original approach...

[Toaster Retribution]

It has always gotten the country right for me, but never the location.

[Sparkrunner]

Appears that it is not embedded or something, all I see are the links.

[Sparkrunner]

(Sorry for the double post) Fixed!

It says "United States" is that intentional vagueness?