Dismiss Notice
How Does This Work?

You create a free account and set a Gmail filter to forward your Ingress email damage reports.

The damage reports are processed in realtime into the Outgress analytics and alerting engine.

That's the short version.

Is This Allowed?

We believe so, but you decide.

But My Secret Data!

No one on the opposing faction cares who on their team is attacking you. Do you care who on your team is attacking opposing portals?

Resolved bug during very large historical import

Discussion in 'Bugs' started by rawr, Jul 26, 2016.

  1. rawr

    rawr New Member Enlightened L16

    I've started the historical import several times now, as it started I saw some of the numbers in the "Your Statistics" box, over in that column, change, but then, eventually, most of those numbers went back to zero, and I had no pie chart for attacks in the Top 20 Attackers in the last month when I hit "Just your data"

    I figured something had gone wrong, so I started another import

    Same thing happened, I saw numbers for a while, but eventually everything went to zeros again.

    Doing it for a third time now... any input?
     
    rawr, Jul 26, 2016
    #1
  2. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    When was the last time you had a portal attacked that resulted in an email damage report being forwarded?
     
    digitalpoint, Jul 27, 2016
    #2
  3. rawr

    rawr New Member Enlightened L16

    I also just set up the forwarding, yesterday, and several emails should have forwarded, and some more overnight. Last ones around 7:55pm Central
     
    rawr, Jul 27, 2016
    #3
  4. rawr

    rawr New Member Enlightened L16

    seems my data from 7/25/13 to 8/5/13 is sticking, everything else is being tossed? (I could search back through my emails to see if it's ALL the data from those dates, or just some, if you need).

    I see that those are back when the format of the email was different, or at least had a different subject line

    Also, I had an agent name change go through sometime around then.
     
    Last edited: Jul 27, 2016
    rawr, Jul 27, 2016
    #4
  5. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    That's probably what the issue was (the name change) and it's a self-correcting issue the first time a new email comes in (that's why I was asking when the last one was forwarded to the system was). When importing emails, it goes in reverse order (newest to oldest) so that if there is an issue, it can pick up where it left off. Because of that, the oldest emails are the ones imported last... and because of the way the system routes emails into the processor, it doesn't make a differentiation between one being forwarded and one being imported.

    Anyway... long story short is with an agent name change, an import sees the old agent name *last*, which the processor in turn thinks is the most up to data info about what your agent name is (since that was the last email it processed from you). It ends up fixing itself because as soon as a new forwarded email comes in, it should make the correction for what your *current* agent name is.

    That being said, from the looks of it, it appears it already made that correction on its own. Let me know if that's not he case and I'll go digging...
     
    digitalpoint, Jul 27, 2016
    #5
  6. rawr

    rawr New Member Enlightened L16

    Ok, yeah, one came in this morning and I see more accurate numbers again. Thanks. Not that I'm worried about the old, old, data, but did that get imported too? The stuff pre-July of 2013?
     
    rawr, Jul 27, 2016
    #6
  7. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    Ya it would have all been imported, it just doesn't know who that data belongs to since it's based on agent name and a single email can only be associated with one agent (so it associates you with the newest agent name it knows goes with that email). There is an admin-side function I can do which will merge the data from your old agent name into the new one if you want. Just would need to know what your old agent name was.
     
    digitalpoint, Jul 27, 2016
    #7
  8. rawr

    rawr New Member Enlightened L16

    Oh, sure, I just thought since it was only showing me that short window of results (until it corrected itself) that it hadn't imported much pre-July of 2013. Emails changed format a couple times but I have emails going back to at least Jan '13

    Previous name was EFDisaster (now DisasterRawr)
     
    rawr, Jul 27, 2016
    #8
  9. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    Data has been merged... Give it a little bit to fully reflect on your stats. The servers cache a lot of stuff on the backend since (normally) historically data doesn't change. So could take up to an hour for the caches to be fully reset.
     
    digitalpoint, Jul 27, 2016
    #9
  10. rawr

    rawr New Member Enlightened L16

    Fantastic!
    I see the data from Jul-Aug of 13 is in my Stats page, now.
    No go on the earlier stuff, eh? Not a big deal to me at all, but if you ever want me to attempt some other kind of export of it, let me know.
     
    rawr, Jul 27, 2016
    #10
  11. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    Nah... The stuff before that was the really old email format that isn't just a different format. There is just a lot less info in those ones. So even with a different parser, there is a whole lot of missing info in those (like if I remember correctly, you can even tell the faction of attackers, which in itself causes a problem right there).
     
    digitalpoint, Jul 28, 2016
    #11
    rawr likes this.
  12. rawr

    rawr New Member Enlightened L16

    That specific problem would only between May and whenever the new format came out (faction of attacker; that's when flipcards came out)... but I hear you, lots of differences in the formatting.

    Excellent work here, love the data and the visualization.
     
    rawr, Jul 28, 2016
    #12
  13. digitalpoint

    digitalpoint Administrator Staff Member Illuminator Enlightened L15

    It wasn't just faction... there were a lot of other bits of missing info that Outgress uses. I don't have access to look at one right now, but if my memory serves me correctly it was:
    • Missing faction (which just in itself makes them useless because Outgress heavily uses that info)
    • No info about broken links (not a deal breaker there)
    • Missing links to intel map for subject portals (no way to know which portal it was because portal name isn't unique... so we can't get longitude/latitude for attacks or portals)
    When it came down to it, the net result of the old email formats was that you know that someone was attacking a portal somewhere. But you can't answer the question about what faction it was or even where the portal itself was. Pretty much makes those old emails useless.
     
    digitalpoint, Jul 28, 2016
    #13