Dismiss Notice
How Does This Work?

You create a free account and set a Gmail filter to forward your Ingress email damage reports.

The damage reports are processed in realtime into the Outgress analytics and alerting engine.

That's the short version.

Is This Allowed?

We believe so, but you decide.

But My Secret Data!

No one on the opposing faction cares who on their team is attacking you. Do you care who on your team is attacking opposing portals?

The Technology Behind Outgress

Discussion in 'Support' started by digitalpoint, Mar 9, 2016.

  1. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    A few people have asked me about what sort of backend technology Outgress uses. So here goes...

    Hardware

    There are a bunch of servers that make up the cluster that run Outgress. The servers themselves are running openSUSE Leap. The servers are interconnected with 56Gbit Infiniband. Each server has the following: a ton of disk space configured in a RAID-6, 512GB of RAM and 32 Xeon CPU cores.


    Software

    Web Server: Nginx
    Application language: PHP
    Local Cache: Memcached
    Search: ElasticSearch
    Mail server: Postfix
    Geo Spatial storage/queries: Totally custom system built from scratch for this project because nothing out there was good enough or fast enough
    Relational database: MySQL Cluster (not to be confused with MySQL). See this video for an overview:


    Other

    CDN: Cloudflare

    That's more or less it... I guess this thread can be used for anyone who has any questions about things or are just curious.
     
    Last edited: Mar 9, 2016
    digitalpoint, Mar 9, 2016
    #1
    batteryAClD and iGoTcHa like this.
  2. wickidd

    wickidd New Member Enlightened L16

    Kinda surprised to see a new site like this using php. Is that due to xenforo? Kudos on the idea for the site and how well it already works!
     
    wickidd, May 16, 2016
    #2
    kldreams likes this.
  3. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    Nah, if I was going to build it with something else, I'd just use a different framework. :)
     
    digitalpoint, May 16, 2016
    #3
  4. methchild_dildrop

    methchild_dildrop New Member Enlightened L15

    curious that I should have found this site, same idea as mine but despite me starting it some 8 months before, I haven't developed mine anywhere near as well as this.
    Locally there are 30 ENL players feeding me their attack reports and the reach / coverage I get is about 60 miles in each direction which isn't too shabby. I didn't scale mine up any further mainly due to the reluctance of people to forward their attack reports, either it was too difficult, they weren't confident doing it or they couldn't see the benefit of the intel provided. We make a secondary game from the stats produced with daily and weekly scores posted to the linked Slack and Telegram webhooks.
    I feel that the time I am able to dedicate to my site means that (although it is ENL only and as such the intel is totally in-house) it won't have much in the way of features added, couple that with the cheap hosting I use limiting things somewhat and I think that the way forward for our local players may be to either move lock-stock over to outgress else for them to set up dual forwarding and share the wealth.
    I will be feeding back to them on my experience here and hopefully you will be getting another 30 or so new sign-ups by the weekend.
     
    methchild_dildrop, Jun 2, 2016
    #4
    digitalpoint likes this.
  5. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    The system can accept data that originates from players who don't set up Outgress accounts (it's intentional because some people just want to help out their local players but don't use the site itself). So as long as the emails were coming to a central place to you unaltered, you could actually forward a forward without any issues (probably still a good idea to let them know/ask them if that's okay with them).

    Also, if you have the historical emails archived in Gmail, the system will let you import them to have all that history instantly.
     
    digitalpoint, Jun 2, 2016
    #5
  6. methchild_dildrop

    methchild_dildrop New Member Enlightened L15

    my system was designed to be closed from the start and so I have players forward their emails to a personalised email address, it helps with filtering out messages not intended to be processed and those that might deliberately spam the server.
    I guess yours won't allow me to backfill until I actually get an attack report of my own , might be a while, I haven't much deployed at the moment.
    I have some 250k attack reports that would potentially be of use, unfortunately they aren't in their original format once forwarded and will probably fail your sanity check.
    I don't know if it would be worthwhile you importing that many but if you wanted to I could set up some way for you to access either the emails or the extracted data.
     
    methchild_dildrop, Jun 2, 2016
    #6
  7. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    Ya, once the system gets a single forwarded email, you will be given the option to import (it's intentional because users might import without forwarding then and then wonder why future damage reports weren't showing up). If you get impatient, you can go ADA your own portal (some people do that... Lol).

    If the emails are forwarded via Gmail forwarding/filtering, the email should still be in its original form (it's the email body that needs to stay intact... The headers don't). If that's not the case and they were forwarded with some method that does alter the email body, I could probably help you still get them into the system if you want.
     
    digitalpoint, Jun 2, 2016
    #7
  8. methchild_dildrop

    methchild_dildrop New Member Enlightened L15

    i'm just seeing if my email client can 'redirect' unaltered reports, if not i'll explore gmail web page. i am too far away from any portals to be going out to ADA it. as much as I love the game i'm not going out especially for that when I can smell that my pizza is in the oven ;)


    was attacked about 29 mins ago so no need to jump through hoops I guess, the import old should now be showing
     
    Last edited: Jun 2, 2016
    methchild_dildrop, Jun 2, 2016
    #8
  9. Richard Quadling

    Richard Quadling @ArrayQueue Enlightened L16

    I wonder if using something like AWS (maybe the Docker Container service) would have been a better (and cheaper) option than hosting your own physical hardware. My work recently moved from a physical box to AWS. Made a HUGE difference. Depending upon your load, you may actually qualify for the free tier, so it'll cost you nothing! Happy to chat more about this if you want.
     
    Richard Quadling, Dec 23, 2016
    #9
  10. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    The AWS free tier has too much stuff that expires after 12 months to make it actually free (we wouldn't qualify for even the 12 months free).

    The servers already were in place for unrelated projects that wouldn't work on AWS (for example we need the ability to pass more than 10Gbit between nodes [which is why they have 56Gbit interconnects]). We also have a bunch of custom helper daemons (including for Outgress that get tricky to deploy properly the way instances spin up/down).

    AWS is nice for a lot of things, but definitely not a "catch all"... otherwise you would see every company with the overhead of managing their own entire data centers using it (Google, Facebook, Apple, etc)
     
    digitalpoint, Dec 23, 2016
    #10
    Richard Quadling likes this.